The British-American Project for the Successor Generation, to give it its full title, was founded in 1985 "to perpetuate the close relationship between the United States and Britain" in the words of BAP's slim official history, through "transatlantic friendships and professional contacts". It has a membership of "600 leaders and opinion formers", drawn equally from both countries. It holds an annual conference (the next starts this Friday in Chicago) to which journalists are not invited and at which everything said is, officially at least, not to be repeated to outsiders. It rarely features in the mainstream media - instead, it makes tantalisingly vague and fleeting appearances in those corners of the internet where conspiracy aficionados gather.

Here, BAP is portrayed as a Trojan horse for American foreign policy, recruiting Britons of liberal or left-of-centre inclinations and political talent and connections when they are young, indoctrinating them with propaganda about the virtues of American capitalism and America's role in the world, and then watching them approvingly as they steer British politics in an ever more pro-Washington direction. According to this analysis, the project's greatest success has been New Labour.

Besides the names mentioned in BAP's 1997 newsletter, the organisation numbers among its members Douglas Alexander, the precocious Foreign Office and trade minister; Baroness Scotland, the politically favoured criminal justice minister; Julia Hobsbawm, the prominent public relations executive and New Labour associate; and Adair Turner, one of the government's most senior business allies and author of the recent official report on the future of pensions.

In the years immediately before the founding of BAP, the early 1980s heyday of Tony Benn and CND, the Labour party was sceptical about America. Now it will seemingly swallow almost anything the US does, and the idea that BAP made the difference has some authoritative backers. The leftwing journalist John Pilger, who has been uncovering American manipulation of other countries' politics for decades, has described BAP as a "casual freemasonry" and "by far the most influential transatlantic network of politicians, journalists and academics". The historian Frances Stonor Saunders, who has written extensively about the American use of earlier, similar networks to influence western opinion during the cold war, sees close parallels with BAP: "All that's changed is that BAP are much more sophisticated."

In December 2001, in response to a parliamentary question from the Liberal Democrat MP Norman Baker, Tony Blair said that the organisation "arranges meetings, including with ministers, for young leaders from the business, economic, professional, cultural, artistic, governmental, academic, scientific, medical, military and social life of the two countries". Beyond New Labour, the BAP membership includes the Conservative election strategist Steve Hilton, the shadow work and pensions minister and Tory intellectual David Willetts, the former Conservative minister Stephen Dorrell, the founder of the UK Independence Party Alan Sked, and Charles Moore, the former editor of the Daily Telegraph.

Until now, BAP's public response to allegations of political influence has been to ignore or dismiss them. A postscript to its official history calls the idea of the project as a vehicle for the American government a "myth" and "a curious reinvention of BAP history". But what, then, does BAP do exactly? Since 1985, it has received sponsorship from, among other companies, Coca-Cola, Unilever, Monsanto, Saatchi & Saatchi, Philip Morris, Coopers & Lybrand, American Express, Apple, British Airways, BP, Cadbury Schweppes and Camelot. Busy politicians and other public figures have crossed the Atlantic, some of them repeatedly, to attend BAP conferences, which can last for five days. One member describes proceedings as "a quasi-religious experience for some people", but what else has kept the whole enterprise going for almost 20 years? What has BAP achieved?

The author of both the project's official history and the article in its newsletter about New Labour is a British journalist called Martin Vander Weyer. He has been a BAP member since 1994, and until last year was chairman of its British operation.

Meeting him, at first, is something of a disappointment. He is disarmingly jolly: amused eyes, a raspy, confiding voice, swept-back grey hair rebelling behind his ears. He is wearing an ostentatiously traditional but slightly unkempt suit of the kind favoured by middle-aged Tory journalists, and has just come back from a lunch at the Spectator, where he is an associate editor. He suggests a cafe, and strides off, talking freely, through the London rush hour. He does not look much like a New Labour conspirator.

Vander Weyer depicts BAP in altogether more relaxed terms. "It's both a fantastic social opportunity and an amazing professional networking opportunity." At the conferences, he says, "Everyone is on equal terms, and you take the handbrake off ..." He grins. "There's quite big late-night drinking. Requires a lot of stamina. Every year you can see the astonishment of the church-going Americans. You see them jogging around the hotel whenever you open your curtains in the morning."

To see anything sinister in all this, he continues cheerily, is "bonkers conspiracy stuff". But what about his newsletter article? Vander Weyer clasps his forehead in mock-regret. "I wrote the headline. I thought it was quite snappy. It was a great mistake. Probably my greatest mistake."

But then he begins to choose his words more carefully. "The British membership is quite a concentrated elite," he admits. "There was a stage where ... a lot of the people who emerged as part of the New Labour leadership group happened - and I say happened, because it is partly chance - to be members of BAP ... The American side is more spread out: Americans who just enjoy contact with Brits. We have Republicans, Democrats, people who work on Capitol Hill."

He explains how BAP members are selected. Each existing member can nominate up to three people aged between 28 and 40. These nominees are then interviewed and tested: there are competitive debates, "management games" and personal presentations. "We sift the nominees according to their willingness to listen to other people," says Vander Weyer. "Whether we think they'd fit with the group."

Do they ever pick people who are anti-American? "Oh yes. There are lots and lots of members who are anti-American." He mentions the journalist Yasmin Alibhai-Brown. Then he grins again. "I've never found myself in such a leftwing group as this." If BAP has a political diversity problem, he says, it is rather different from the one its critics allege: "Some of the drier Conservatives who've come to conferences over the years have found ... too much of a bleeding heart."

The circular nature of the nomination process narrows the BAP membership in other ways. A member recalls, "I was nominated by the man who subsequently became my husband, who was nominated by a friend of his, who was nominated by someone he knew." Vander Weyer says BAP is trying to compensate for this. "We try to find people outside our network." He cites "a cinema manager and a fire officer from Newcastle" who have become members. He gives a rare serious look. "We want to counter any sense that this is a self-perpetuating elite."

The only problem is, BAP was founded to be exactly that. At the start of the 1980s, the idea of a "successor generation" began to stir on the dusty pages of British and American foreign policy journals. Kenneth P Adler, an academic employed by the US government to watch western European political trends, defined it as "the segment of the general public that is most likely to succeed to positions of power and influence in the near future". This group, he and other observers predicted, would either follow the broadly pro-American path of "the founder generation" of postwar western European political leaders, or take a more independent, even hostile stance. With Ronald Reagan in the White House and the cold war he was helping to orchestrate in one of its least appealing, most attritional phases, it was by no means certain "the successor generation" would stay loyal to Washington. "Ideologically, if a consensus exists across Europe," wrote Adler, "it would be somewhere on the left ... a middle way between Sweden and Yugoslavia ... distancing [itself] from 'the superpowers'."

The US government was particularly worried about Britain. Despite all the official talk of a "special relationship" between the two countries since the second world war, there had been surprisingly regular periods of British public disenchantment with Washington: over Suez and Vietnam and, particularly, over the deployment in Britain of US nuclear weapons. In November 1981, three weeks after CND had held its biggest ever protest in London, Reagan made a speech in Washington warning that "some young people do not understand ... why we need nuclear weapons [or] Nato's roots in defending freedom." With Margaret Thatcher a deeply unpopular prime minister, and the Labour opposition influenced by an anti-Washington party membership, a new British official attitude to America looked quite likely. "It is possible to argue that had a Labour government been formed," the historian Peter Jones wrote of the early 1980s in his book America And The British Labour Party, "it would almost certainly have led to the complete collapse of the 'special relationship'."

A 27-year-old British economist called Nick Butler decided to intervene. For someone of his age and profession, he already had unusually useful and diverse connections: he worked for BP, but he was also treasurer of the influential left-leaning pressure group the Fabian Society and a promising junior player in the Labour party. He also loved America. "The UK was in a bad state," he says. "America seemed much more dynamic, full of ideas, open." For years, he had been reading Tom Wolfe and Joan Didion, and books about US politics. He was always wanting to visit America, more than his work permitted. At the same time, he felt the Labour party needed fresh ideas from abroad. "My perspective then was that my generation - I would have been described as 'rightwing' in the 1982 Labour party - were totally stifled here. No prospect of being in power."

That spring, Butler wrote a memo proposing "some form of regular contact for Britons and Americans", to reduce "hostility to all things American" and promote "mutual understanding over a wide range of policies ... how cities are regenerated, how market forces worked, and so on." For the membership of what was to become BAP, he had a specific transatlantic group in mind: "Bright people, in many different fields, who were likely to influence outcomes in those fields. People who were interesting. Interested in change. In doing things. In progress."

Butler says this with a straight face and takes a sip of wine. Sitting in the half-darkness of his London club, wearing an immaculate suit and barely opening his mouth as he talks, he explains the philosophy and workings of BAP with less of Vander Weyer's cheery evasiveness. "I don't think networks are inevitable," Butler says with emphasis. "They are absolutely desirable. I think networks are a great phenomenon. The Fabians are a network, BP's another network - that is civil society."

Since the 1980s, Butler has maintained and extended his political and commercial connections like a model member of the "successor generation". He is close to Mandelson and other senior New Labour figures. Thanks in part to Butler, BP - where he is now group vice-president, strategy and policy development - has become known as "Blair Petroleum" for the warmth of its relations with Downing Street.

Butler's gifts for alliance-building and persuasion turned BAP from a paper proposal into an international organisation in less than three years. Between 1982 and its first conference in 1985, he recruited a shrewdly broad range of supporters, co-founders and financial backers: Sir Charles Villiers, a liberal Tory businessman with a long personal attachment to America; the US embassy in London, which gave Butler a grant to go to Washington to test reactions to the BAP idea; and the Pew Charitable Trusts, a very large and wealthy American foundation.

These days, Pew supports diverse causes, from public health to the environment. But the foundation's origins are more controversial. The Pew family made their money in oil, and for much of the 20th century the dominant personality in their business and philanthropic activities was J Howard Pew, a man of particular political convictions. "To me free enterprise is a very noble thing," an official Pew history from 1984 quotes him saying. During the 1950s, he established the J Howard Pew Freedom Trust to, in the words of its charter, "acquaint the American people with the evils of bureaucracy ... the false promises of socialism ... the paralysing effects of government controls". Since his death in 1971, the official history continues, the Freedom Trust "has supported those projects and groups that reflect the founder's philosophy". BAP, it appears, was one of them: according its own official history, it received "grants totalling $460,000, which funded the first three BAP conferences".

Butler says that the Pew organisation "never interfered. Never told us what to do. I never met them." He sounds convincing. Yet, if you read the reports from these and subsequent BAP conferences, written by and circulated to BAP members, and talk to some of those who attended, a process of political education can be discerned of which J Howard Pew would have approved.

Every autumn, BAP hires a hotel, or a large part of one, for a long weekend, alternating between British and American venues. Conference rooms are reserved, boardroom-style tables arranged, themes chosen for discussion. A purposeful timetable of seminars and larger gatherings, dinners and group excursions is drawn up. Lighter interludes are scheduled for drinking and bonding and organised fun - Vander Weyer has been known to host a closing-night revue as a rightwing caricature called Professor Whiplash - but the overall atmosphere remains somewhere between an international summit and a corporate retreat for young executives. Even at the more intimate seminars, there are papers and water jugs on the tables, and some of the men like to keep their ties on.

"I've been on weekends organised by other networks - Anglo-French, Anglo-Spanish, Anglo-German - but I've never been on such a grand one," says Alibhai-Brown. "The amount of drink, the way you were treated, the dinners with everyone who was anyone ... Jonathan Powell [Tony Blair's chief of staff] used to come a lot. I remember having many an argument with him beside swimming pools in white towelling dressing gowns ... It was money that I'd never seen at any conference before. We [the participants] used to joke, 'This is obviously funded by the CIA.'"

Any such connection to Washington is denied by BAP, but a more subtle subordination to America was there from the beginning. There was the fact that BAP started as a British initiative, not an American one. And there was the way the American members set the tone of the conferences. "I vividly remember the first: we were all stunned by how much more money they all had," says Butler. "They'd run their own businesses. They ran charities. They were in a different league in terms of what they'd achieved. I remember Jeremy Paxman saying that to me, and Mo Mowlam. We were all struck by that."

This sense of American superiority framed and coloured the discussions. As people leaned forward on the conference tables and made bright-eyed presentations and asked each other penetrating questions, European notions such as socialism, the welfare state and high levels of government spending were judged, in the slightly sweeping way of clever young thinkers, to be in difficulties. American notions such as less regulated capitalism, a smaller "enabling state" and a world kept safe by the Pentagon came to be regarded as sensible, inevitable. "Five years before I joined BAP, I thought wealth creation and progressive politics were completely incompatible," says Trevor Phillips, now chairman of the Commission for Racial Equality. "BAP was one of the things that made me think that was absurd."

For the American participants, the political epiphanies were less dramatic. "Americans who might have thought that the Labour party was off the radar screen had their minds opened," says George R Packard, a prominent American international relations academic and BAP supporter from the beginning. Another American member says simply, "You learn a lot about how many perspectives there can be." In Phillips's experience, the traffic in specific ideas was mostly one-way: "I didn't detect that the Americans had learned a great amount [from us] about what they could and should do in the US."

Besides the advantages of the American way, the other big political preoccupation of BAP conferences during the 1980s and early 1990s was the condition and prospects of the Labour party. The Conservatives may have been in government, but the conference reports mention them surprisingly rarely. Instead, there are involved discussions about the Labour attitude to nuclear weapons, about the strengths and weaknesses of the party's general election campaigning, about whether, as BAP members saw it in 1987, Labour "had more interest in remaining a party, with policies and an ideology, than in achieving power".

It is too conspiratorial to see in these debates the creation, in secret, of New Labour. The remade party was the product of countless gatherings and discussions, and too many of its architects - Blair and Gordon Brown, to name two - were not BAP members. But, in retrospect at least, there was a whiff of something new in the analyses of Labour's problems presented at BAP conferences. "Most of us held broadly New Labour views - before Tony, you could say," says Lord Lipsey, the journalist, Labour peer and BAP member. "BAP was one of a number of streams that came together in New Labour."

Of course, not everyone in BAP is a politician, just as not every conference seminar is about politics. "For any 10 politicians who happen to be members," says Vander Weyer, "you could name 10 artists, writers ..." He pauses, and then a gleam comes into his eye: "Benjamin Zephaniah [the radical black British poet] is a member. Although not a regular attender."

Another element of the BAP membership is less surprising. "Many BAP alumni are directly involved with US and UK military and defence establishments," noted the 2002 conference report. An account followed of a conference excursion to the Pentagon: "Our BAP group was welcomed as 'old friends'." Butler is equally frank about the link. "The military are quite important, quite influential. We had a great guy who was a Polaris submarine commander. And he was a leftwinger ... loosely leftwing. [Colonel] Bob Stewart, of Bosnia fame - a lovely guy - gave a great break-out talk. These weren't people pumping out a military line. These people were talking about their direct experience."

The encounters and contacts that BAP makes possible are often cited enthusiastically by members. An American mentions meeting an astronaut at a conference. "We kept in touch ... then he told me he was giving a lecture at West Point [the US military academy]. I took the train up and we had dinner. It was a blast! Completely out of my normal world."

Packard says that at the conferences he is "astounded at how quickly the bonding takes place". In BAP's official literature, the former Labour minister Chris Smith describes attending one as "four intoxicating days of thought and discussion".Critics of BAP say that is precisely the point. "Propaganda that really works," says Saunders, "is when you get people to move in directions you want them to for reasons they think are their own."

Yet not everyone joins BAP with their guard down. "I did make some inquiries privately before joining," says the British writer and foreign policy analyst Anatol Lieven. He cites the infamous British liberal journal secretly funded by the CIA during the cold war: "After the whole Encounter experience, one does have to be a little careful." But Lieven's inquiries about BAP left him reassured: "It is genuinely pluralist. The discussions are very frank. In 2002, they asked me to give a talk on Bush's strategy in which I was very, very tough."

Other critics of Washington join BAP in order, they say, to know their enemy. Alibhai-Brown found her first conference in 1988 "a miserable experience ... but really useful. There were hardly any women, and unspeakable people - so rightwing - on both the British and American sides. But I wanted to know about this very powerful axis, to learn to talk to them without poking them in the eye." She was still attending BAP conferences 15 years later.

A certain number of internal dissidents are good for the project's image: they make BAP, and the Anglo-American relationship, seem open to criticism but too important to ignore. And they keep the conferences interesting. But, reassuringly, not every rebel is successfully co-opted. Zephaniah recalls his one and only conference: "It was in this hotel in California, in Oakland, the Claremont. I remember them [the BAP members] all as men in suits or power-dressed women. Oil people, a couple of people from minority groups. I remember loads of trust games. The men were told, 'Now take off your tie, and relax, and do some yoga exercises, and go off into a group, and talk about empowerment.'"

Zephaniah started skipping the discussion groups by telling each one that he was going to the other. But after a while he had had enough. One evening, "I escaped. I got out of the hotel. I went down to Berkeley [the neighbouring city], hung out with some homeless people, went to see a friend of mine." How did BAP treat him after that? "Every year, they kept sending me the report of the last conference. I had a whole shelf of them. Last year, I put them in the bin."

Sitting in his London club, in his immaculate suit, Butler smiles. We have been talking for perhaps 20 minutes, and we are already on to the second glass of wine. The founder of BAP, like many of its members, is good at being convivial. "BAP is not a terribly serious venture," he says. "It's an interesting venture." He says he feels "quite proud" of what it has achieved. "A lot of people have learned a lot from American experience in a lot of fields."

He says he hopes that the British members of BAP can exert a moderating influence on America in return. Perhaps. The more liberal American members of BAP also hope so. But, as with the special relationship itself, the power and uniqueness of BAP can be overstated. There are other Anglo-American networks for the young and ambitious: Rhodes scholars, Fulbright scholars, Kennedy scholars. And there are other American networks. Packard mentions one in passing: "In 2000, I started an exact clone of BAP: the US-Japan leadership project."

Behind the confidence of the BAP conferences, according to Vander Weyer, lies a skeletal organisation: no British office, a "cubicle office" in the US, a tiny staff working from home. "From time to time we receive a small amount of money from the foreign and commonwealth office, the US embassy," he says. After getting their first conference for free, members pay up to ?500 each to attend.

And every year, BAP needs new members. As he gets up to go at the end of our interview, Vander Weyer gestures expansively across the cafe table. "Depending on how this goes," he says, "I'd be very happy to nominate you."

UK members of the British-American Project include:

Peter Mandelson EU trade commissioner
Jonathan Powell Tony Blair's chief of staff
Jeremy Paxman broadcast journalist and author
Mo Mowlam former Labour Northern Ireland secretary
Adair Turner head of pensions commission
Trevor Phillips chairman of the Commission for Racial Equality
James Naughtie broadcast journalist and author
Matthew Taylor Downing Street head of policy
Chris Smith former Labour culture secretary
Baroness Symons Foreign Office minister
Lord Robertson former Nato secretary-general
Douglas Alexander Foreign Office and trade minister
Geoff Mulgan former head of Downing Street's policy and strategy unit
Baroness Scotland Home Office minister
Julia Hobsbawm public relations consultant
Steve Hilton Conservative special adviser
Benjamin Zephaniah poet and activist
Colonel Bob Stewart former commander of British forces in Bosnia
David Willetts Conservative shadow work and pensions secretary
Alan Sked founder of Ukip
Stephen Dorrell former Conservative health secretary
Yasmin Alibhai-Brown columnist and broadcaster
Charles Moore former editor of the Daily Telegraph
Nick Butler BP group vice-president, strategy and policy development
Lord Lipsey Labour peer and author