The Numbers Game: Transparency and Reliability Issues in Estimating Russian Rebel Involvement in Syria and Iraq

Articles discussing rebel involvement in the conflict in Syria and Iraq and the threat posed by returnees often cite estimates by the Soufan Group. In this research note, I’m going to explain why this is problematic in relation to Russia, and identify some of the challenges in estimating how many Russians have travelled to the Middle East.

The Soufan Group has become one of the go-to sources for estimates on how many foreign citizens have participated in the conflict in Syria and Iraq. Its reports attract significant media attention and shape both public and academic debates, and they are often used to support arguments about the potential threat posed by returnees.

The group’s “findings” are typically presented as if they are an independent estimate. Following the publication of its latest report in October 2017, for example, RFE/RL reported:

Two years ago, the Soufan Group estimated that Tunisia and Saudi Arabia were the largest sources of fighters in Syria and Iraq, followed by Russia.

Now, the group says, more than 8,700 from the former Soviet Union have travelled to the region, and an estimated 3,417 from Russia alone, primarily from the troubled North Caucasus, where Chechnya is located.

RAND analyst Colin P. Clarke similarly testified to a US House Foreign Affairs subcommittee that:

A new report by the Soufan Group estimates that Russia is indeed the largest exporter of foreign fighters to the conflicts in Iraq and Syria, with more than 3,200 fighters.

There are many such examples.

The problem is this: The Soufan Group does not have any independent data on how many Russians are involved in the conflict in Syria and Iraq. It is entirely dependent on Russian official figures. If you are citing the Soufan Group on the involvement of Russian citizens, you are citing a Russian government agency or official, not an independent estimate. Experts in other areas of the world need to assess whether the same is true for their areas of interest. But, given that Russia is presented as the single biggest contributor, this certainly impacts the group’s overall estimates of foreign citizen participation.

What’s in the Report?

Russian rebel involvement in the conflict in Syria and Iraq is mentioned at several points in the report. On page 10, Russian President Putin’s claim that 10% of 9,000 people from Russia and the Former Soviet Union have returned is mentioned. This is clearly an official statement, and is presented as such. Below that, a graphic shows the top five contributing countries, with Russia the highest at 3,417 – the figure cited by RFE/RL. A footnote unhelpfully observes “these figures are an approximation based on available sources.”

On page 13, a table repeats the 3,417 figure, this time sourced to articles in Meduza and Rossiyskaya Gazeta. The former is in any case citing the latter, and Rossiyskaya Gazeta’s figure comes from a March 2016 statement by the Russian Interior Ministry. In other words, the Soufan Group’s widely cited “estimate” is actually an Interior Ministry figure. A further reference observes “informal estimates by security services estimated the number at over 5,000 in July 2017.” Here, no source is provided, but it is also clearly official. The table also reports 804/4,128 as “sentback/stop listed in Turkey” – this time apparently based on a Turkish government report – and 400 returned, again with no sourcing.

On page 25, a table on “foreign women and children in the Islamic State” repeats the 3,417 total figure – despite the fact that Russians have been represented across a range of rebel groups, not just the Islamic State – and claims 350 of these are children. This latter figure is sourced to another Russian official, this time Anna Kuznetsova, presidential ombudswoman for the rights of children.

Thus, all of the Soufan Group’s figures for Russia are based on official estimates. One can argue about whether they are sufficiently transparent about this or not. On the one hand, all the information I’ve just cited is in the report, so – some slightly woolly sourcing aside – they provide the means to critique it. On the other hand, the Soufan Group package the information as original research, and they hardly push back against those who treat it as an independent source.

And why should they? The Soufan Group are a business, and their business is selling security and “intelligence.” As long as there are customers, there will be dealers. All appearances to the contrary notwithstanding, this is not an attack on the Soufan Group.

Inconsistency and Uncertainty in Russian Official Reporting

Does it matter who produces these figures? Estimating how many people have travelled to Syria and Iraq is, after all, more of an art than a science. Well, I would argue that transparency is an essential part of rigorous research, and we can only properly scrutinize information if we trace it back to source. Otherwise, we risk getting stuck in a stale and flawed information loop.

And closer examination of Russian official statements illustrates some of the pitfalls of relying on single estimates in isolation. The following graph plots federal-level official statements since the start of the conflict:

Screen Shot 2018-04-05 at 02.41.39

What this graph shows is that different agencies can offer different figures, and even the same agency can provide contradictory information. Moreover, when statements are examined in detail, it’s often unclear what time periods are covered and whether figures referring to the Islamic State are actually referring to the group, or part of a narrative that treats all groups as the Islamic State.

Russia also faces a number of distinct challenges in arriving at these estimates. Russian citizens who have travelled directly from Russia are only one possible group. There are also Russian citizens who have travelled from student and work communities in Turkey and the Middle East; refugees from the conflict in the North Caucasus who may have renounced their Russian citizenship and/or obtained residency and citizen rights in other countries; members of associated ethnic groups, such as Georgian Kists; and non-Russian citizens who have travelled from Russia, such as Central Asian migrant workers. Even without a political agenda – and this is hardly a given in Russian official figures – disaggregating these groups is challenging, if not impossible.

Yet the differences matter. If individuals are foreign citizens or have the right to remain in other countries, obtaining information on them can require international intelligence cooperation, and all the difficulties that entails. If individuals left the North Caucasus years ago and have failed in previous attempts to return, this is highly relevant to considering the potential threat posed by “returnees.”

For this reason, it’s important to be clear on the limitations of the information that we have at our disposal, and to provide the necessary caveats – however unfashionable ambiguity and uncertainty may be. The further we go from the original source, the harder that becomes. Critical examination of what we know and how we know it is essential if we are to avoid building our arguments on sand.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s