Most people only see a conductor at work during the performance, but most of their work has been done in the rehearsal, explaining sometimes verbally, sometimes in gestures, what is required to achieve their idea of the perfect performance. A good conductor will not just lay down their idea of the orchestra, but will be working with the orchestra to combine their ideas with conductors, work with the strengths of the orchestra and hide the weaknesses, to make a seamless performance for the audience.
We now come to why the conductor doesn't exist in the other ensembles. In a string quartet, each person is communicating with the other 3 what they are about to do, while wishing to do with music. That person is also reading from the others what they are about to do or what they wish to do. It is a collaborative effort amongst 4 people, and they must know who has ultimate say during the performance, at any point in music. The person who has the ultimate say in the quartet changes throughout the piece of music. I conductor would be pointless and ignored. The only time one gets near to having a conductor is during a masterclass, where the string quartet as a whole are coached by the person running the masterclass, who will be a string quartet play themselves, but will be taking a detached view the entire performance. In a chamber orchestra, if it's a small chamber Orchestra, the same applies except that the leader of the first, 2nd violins, viola's and cellos will be playing as a string quartet amongst the 4 of them, whilst the remainders of the sections are following the leader. In this way the whole performance can hang together very precisely. If it's a large chamber Orchestra, there can't be enough contact between all the musicians, and a conductor gets drafted in.
In a jazz band, it is much the same as a classical string quartet, though there will be an overall leader who indicates which instrument is going to take the improvisation lead at any moment in time.
The ceilidh band is much the same as the jazz band or string quartet, except there is in our way, a conductor all conductors (plural). The barn dance band have to play at a speed that is suitable for the dance and the dancers, and sometimes the caller, who will have a better view of the dancers than many of the band members, may indicate a change in tempo. The band are also watching the dancers, though sometimes with stage lighting it is very difficult to see what's going on in the dance floor. If they can see what is happening, the person who is playing the lead instrument will alter their tempo to suit. The speed of a dance depends on many things, the dances of the complications of the steps and interweaving is an obvious one, but not so obvious is the fitness, weight and height of the dancers. If you're playing for members of the gymnastics club, you can normally play the music a lot faster than if the dancers are members of beer swilling clubs. If there are a lot of small children dancing, because of their small weight and physical size, they will naturally bounce up and down much faster than adult, so the music tends to go faster for them. If the dancers are big fat people, then their natural frequency of oscillation will be much lower, you will have to play the music much more slowly. Basically, you are playing music to match the natural bouncing speed of the group of people that you have. A ceilidh band is effectively matching their music speed to the natural bounce frequency of the dancers. If all dancers are of similar age or fitness, then it is straightforward. If there is a wide range of ages, sizes and fitness is amongst the dancers, the barn dance band has to play the slowest answer if everyone is going to be able to dance, but that gets boring for the fitter people. Effectively what a ceilidh band will do is do some dances slowly with the least fit, and do some fast to suit the fit people and hope that the unfit people give up and go and sit down, or perhaps announced that the dance will be done at high speed for those who are particularly energetic.
So we've had a good look at music and the instruments of music, but what about when the music is played, what sort of occasions. Human society has been governed to a large extent by ritual. In mankind's move towards reason from mysticism and the supernatural (from many people, that journey hasn't even begun), mankind has set up a string of magical and mystical observances which in the main were designed to warn off imagined evils, to appropriate gods which mankind made variously in his own image, and communicate praise with Thanksgiving the successful outcome of various major undertakings. The chief amongst these rituals have been associated with birth, circumcision, the giving in marriage, death, walls, harvests and the succession of the seasons. s seems to have gone on throughout the world in every society that has developed.
This process has happened in a very similar manner in societies throughout the world, that have had very little contact with each other. In virtually every society, these events have been accompanied by music. Music, more than any other man-made thing, has the ability to inspire frenzy and sue youth distress. Music for weddings, baptisms and funerals. Music for war marchers, peace marchers, social demonstrations. Music to honour the gods, or God, celebrations of harvests, prayers for deliverance from danger, prayers for atonement of sins, all set to music. All of these social manifestations, the social group activities, still play an important part in the life of many communities. Where it doesn't play a part, music comes into other activities such as films, TV adverts, music downloads, parties and discos. But the original purpose of much of music was the accompaniment of ritual.
When one of our string quartets plays for a wedding, whether it is a civil wedding ceremony or traditional church wedding ceremony, it is adding something special to the ritual of the occasion. When a ceilidh band plays for village ceilidh, it is for a bringing together of the community. The jazz band playing for the silver wedding anniversary, is the ritual of the celebration of a long and happy marriage.
There's always been music for the common people to enjoy, dance music, roving minstrel's playing tunes in market squares et cetera, but the highbrow music was mainly the music of the church up until the 15th century. Still today, there is big business in church music. Also many churches, because of their musical background, kindly host music of nonreligious form, putting on string quartet concerts and even jazz concerts in the churches for the benefit of the local community. We also organise by dance bands or ceilidh bands for many harvest suppers or other church events, so music remains strong within the church.
In its ritual function, be it for the church or to warn off pagan type evil spirits, it was not music to be listened to on an intellectual level while sitting in a concert hall. I suppose the nearest early beginning to the concert hall, was the wandering minstrel who put up for months on in a pub or in village square and collect money from the audience that gathered around them. A lot of early music was more the recitation of some heroic tale, with a bit of musical accompaniment to make it more interesting, and was not music for its own stake. This still happens with the traditional folk song, which of course will cover the subjects of love, but in folk music has probably much wider range of stories and interest than that covered in the pop culture.
The first period in which music to listen to can be said to have established itself was during the 16th century, when madrigals and consorts of viols began to appear. The origin of the word madrigal seems rather unclear, with some musicologists attributing it to Italian, others to Greek and yet others to Spanish. Whatever the origins, it is a music that is descended from troubadours and minstrels who wandered around Europe in the Middle Ages, singing stories and spreading news in song. The earliest madrigals go back to the 15th century, and came from composers who were primarily producing church music (presumably because the church was wealthy and provided them with a living.)
Madrigals became the rage throughout Europe, moving through composers in France, Italy, Germany and England, each developing their own style. While all this was going on, there was a parallel development happening in instrumental music. This was based on the new and very much in vogue instrument, invented in the 15th century, no not the electric guitar but the viol. This was purely music, no words, no stories, no message. English with a Masters of this new art. Names like Tallis (1520 to 1585), Byrd (1538 to 1623), Morley (1558 to 1603, and so on, with the pop composers of the day. This is the beginning of music for pleasure, without which much of the record industry and classic FM would be out of business.