Qualified Lawyers Transfer Scheme Research

3 September 2012

Executive Summary

The regulatory model and the SRA role

Key drivers for the regulatory model

Different regulatory models are deployed in various professional contexts and other legal jurisdictions, driven by context-specific needs and drivers:

• The level of risk: The key variation in approaches to admitting international professionals is in relation to the level of regulatory risk in each case. There are arguably relatively few 'high risk' professions in which the public protection considerations of the regulator lead to a high level of scrutiny of international professionals.
• The type of professional regulation: The regulatory model is also a function of the type of professional regulation enforced within a given jurisdiction or profession (i.e. the basis on which a professional might seek recognition). Where there is no requirement to undergo a professional recognition process in the host country, the professional status that is conferred takes on slightly different characteristics and this can lead to a process with quite extensive regulatory 'hoops' for candidates. Qualified Lawyers Transfer Scheme (QLTS) is an interesting counterpoint here in that the scheme design is notable in its accessibility and relatively low threshold for lawyers wishing to access the assessment (even though the assessment itself is challenging).
• The approach to regulatory reform: Most well-established professions within a country or jurisdiction might be regarded as conservative in nature. The maintenance of professional standards in this context is, in part, based on their perceived impermeability and this creates the conditions for a risk-averse approach to reform. With notable exceptions in the medical field, it means that the basis for confidence in the process for admitting international professionals is past experience rather than objective assessment of the reliability and validity of the process itself. This also means that there is much more ad hoc refinement of admission processes than wholesale reform. There were arguably a fairly unique set of drivers for reform facing the SRA prior to the introduction QLTS (the volume of applicants through the international route, combined with a recognition that the existing system was not fit for purpose and linked to wider reform of the regulatory landscape as set out in the Legal Services Act 2007).
• External factors: different professional entry routes depending on country of origin: There are numerous professions where, in practice, the process for candidates is determined by country of origin. The significance in the context of QLTS is that, based on the regulatory principle of nondiscrimination, the model generally envisages the same route for applicants irrespective of country of origin. This is not always the case in other professions. It might be argued that the absence of different requirements attuned to the specific education and training of applicants may lead to disproportionate entry requirements on some candidates. However, in practice, the existence of different access routes depending on country of origin tends to operate more to exclude than to include. While it is not inherently discriminatory to offer a 'fast track' route for applicants from some countries, there needs to be an objective evidence base for setting these requirements. There are cases that appear to be proportionate, generally applicable and based on robust analysis of different education, training and professional systems, but these are exceptional. It is not realistic to expect many regulators or professions to have the resource to be able to develop this kind of complex, global evidence base.

Outcomes-focused regulation

From a comparative perspective, it may be worth considering two dimensions to outcomes focused regulation:

• The notion that applicants are assessed in relation to competence (achieved learning outcomes) as a basis for professional recognition, usually defined in terms of professional standards. Most professional regulators in the UK assess international candidates in relation to competence. This is arguably less the case outside of the UK, where there tends to be a greater focus on assessing knowledge and inputs to education and training (i.e. number of years or hours of study).
• The idea that the regulatory system itself is outcomes-based – in that it is less concerned about the process by which an applicant has become professionally-qualified (the training and experience inputs) than whether they are assessed as meeting the required professional standards. There are typically a number of steps to any admission process for international professionals, but it is possible to draw out a key distinction in terms of whether the approach is based on individual scrutiny of a candidate's qualifications and experience or on assessment / examination. Neither approach is intrinsically more aligned to outcomes-focused regulation. However, in practice, most regulators adopt some form of combination of the two approaches. Admission processes are therefore on something of a continuum, and it is the relative emphasis on individual scrutiny and/or assessment that determines how outcomes-focused the regulatory model is.

The role of the regulator and resources for assessment

There are two dimensions to understanding the role of the regulator:

• The role of the regulator in providing oversight and quality assurance to the assessment process.
• Approaches to internal or external provision of all or part of the overall assessment process.

The model of regulation goes some way to determining how the regulatory role is likely to be determined. There is something of a trade-off in terms of whether significant resource has to be deployed for the administration of individual scrutiny of candidates or for the development and running of assessments for international lawyers. Where the number of applicants is low, it may not be economically viable to run an assessment – unless there is strong public protection rationale for doing so.

However, it should be noted that numerous legal regulators direct significant resource to the detailed scrutiny of individuals applicants, while also including an assessment element (albeit a less costly and sophisticated assessment). The SRA is, in effect, concentrating its resources on the assessment and, as a result, is able to provide a more sophisticated examination for a similar level of input as those regulators that have a more clearly defined two-stage process (detailed individual scrutiny followed by assessment). On balance, this suggests that the QLTS model constitutes a good balance between safeguarding consumers of legal services and ensuring that resources are effectively directed.

Where assessment is used to admit international professionals, the key regulatory distinction is between those regulators that manage the process in-house and those that use external provision (especially with regard to setting and delivering the examinations). There is much less of a focus for regulators on the number of assessment providers than the question of whether assessment should be undertaken in-house. Very few regulators use multiple assessment providers (and none of the regulators looked at in detail as part of the study). The idea of having multiple assessment providers is not seriously considered because of the perceived regulatory risk. This concern about regulatory risk is also a key driver for maintaining the primary assessment function in-house, even where a growing number of candidates put pressure on internal capacity.

For most regulators, a prime objective is to retain as much control as possible over the process for admitting internationally-trained professionals. However, the scale of resource and expertise required to run complex, high volume assessments in-house is not a realistic proposition for most regulators.

This provides a strong case for having a single external contractor with close oversight from the regulator.

The assessment model

Proportionality and accessibility in the context of QLTS

The QLTS widened eligibility to 80 jurisdictions worldwide (counting the USA and Canada as single jurisdictions), with 27 new jurisdictions that include significant populations and extend the reach of potential accessibility to the profession substantially. To date there have been four rounds of the MCT assessment (and three rounds of the OSCE and TLST). There is a clear trend of increasing numbers of candidates, albeit from an understandably low base (from 25 candidates in MCT round one to around 250 for MCT round four). There is evidence the scheme has been somewhat successful in attracting some applicants from new jurisdictions, but that there is potentially significant latent demand in these countries as well.

Towards the end of the QLTR scheme, it is estimated that there were around 2,000 candidates a year. Generalising from the most recent volume of candidates for the MCT, it might be assumed that the QLTS is currently serving around a quarter of this number of candidates (i.e. around 500 candidates per year). It is difficult to directly compare candidate volumes for the two schemes as the number of candidates would be expected to reduce as a consequence of the raising of the standard. Furthermore, stakeholders generally agreed that the winding down of the QLTR scheme would impact on the QLTS. It is logical that potential candidates who were considering becoming qualified as solicitors in England and Wales around 2010 would gravitate towards gaining the QLTT certificate of eligibility while they could. Wider market changes may also impact on the demand for QLTS, especially in the context of an economic downturn (e.g. anecdotal evidence that City firms have substantially altered their international recruitment strategies in light of the downturn).

It has been argued that the difficulty and increased cost of the QLTS assessment compared to the QLTT has a negative impact on those potential candidates interested in professional and career development. In fact, many of the concerns raised by stakeholders relate specifically to the impact on those candidates who do not wish to actually practise in England and Wales. The assessments may be problematic for specialists in that they may have to reacquaint themselves with areas of law that are long-forgotten. It is not clear that this is a regulatory issue, though. The notion of specialist or partial accreditation raises an additional set of regulatory challenges that would have to be set alongside the rationale of convenience for international specialists. From a regulatory perspective, the risks appear to outweigh the potential benefits (not least because it adds complexity to the system). It is unrealistic to expect consumers to necessarily understand the subtleties of any such partial or specialist system – providing greater risk of confusion and possibly reduced overall confidence in the system. It also adds an additional risk of partially-qualified lawyers operating beyond their remit, which may require additional monitoring from the SRA in a way that sits uneasily with its outcomes-focused approach. It is also practicably impossible to effectively monitor in relation to lawyers becoming partially-qualified in England and Wales and then practising in other jurisdictions on the basis of being fully-qualified in England and Wales. There may be more substantial evidence on this question emerging through the LETR.

Assessment method

The rationale for the QLTS assessment methodology appears to be strong. The fact that it is not more widely used in legal settings says less about the a priori appropriateness of the assessment than the wider constraints that shape the regulatory model in other jurisdictions and in other areas of law. Early published statistical analysis of the performance of the assessments shows high scores on technical quality indicators.

There is a powerful rationale for designing an assessment that tests competence in a comprehensive and objective fashion, rather than defaulting to an additional de facto training requirement (as is the case for the New York Bar and many other legal jurisdictions). The three QLTS assessments provide a sensible approach to covering the day one competences for solicitors. The use of standardised clients as part of the assessment provides an opportunity to test the real-life application and professional skills in a way that is closely aligned with outcomes-focused regulation.

Each of the assessments draws heavily on existing tests, which provides a degree of confidence in their application (e.g. the MCT is modelled on the US Multi-State Bar Exam; the OSCE drew on the PLAB examination). Fundamentally, the focus on competence rather than simply testing knowledge provides the basis for far greater confidence in admitting international professionals. There is recognition of this even among regulators that currently use knowledge-based examinations.

Cost of assessment

The assessment fee for the QLTS assessment totals £3,230 excluding VAT per attempt. Although the QLTS is clearly an expensive assessment, it is comparable with the kind of cost associated with the schemes in dentistry and architecture. Cheaper assessments in medicine are not run on a full cost recovery basis and are cross-subsidised by other activities. For smaller examinations, quality assurance procedures tend to be less sophisticated, but it is not clear that the same level of reliability could be achieved by reducing the preparatory inputs (e.g. training and using a substantial number of actors). A greater concern may be to understand the break-even point for financing this type of assessment – as the QLTS' long-term sustainability is presumably based on having a minimum number of candidates each year. This provides a strong argument for maintaining the current assessment relationship given that development costs are front-loaded and candidate numbers in the first year at least were unsurprisingly low.

Even though the equivalent exam costs for the New York State Bar are much lower than the QLTS (and according to the New York State Bar, the examinations are cross-subsidised), the true cost of admission of international lawyers to the New York Bar is much higher. Most international lawyers are either required or encouraged to undertake a one-year LLM in the US, the fees for which run into tens of thousands of dollars.

Single versus multiple assessment provider models

Having a single assessment provider is the most straightforward way of safeguarding assessment standards. It is easier to ensure consistency in assessment standards with a single provider. It also enables the SRA to have detailed input into (and therefore control over) the design and (ability to respond to) monitoring of the assessments.

There are also risks to introducing competition within the assessment market if it is assumed that assessment providers compete on elements that are potentially counter to the need to safeguard standards (e.g. pass rates, costs). Whether this is a valid concern or not, there is a significant and widely-recognised risk of competing providers leading to negative perceptions of the credibility of the exam. Stakeholders in other jurisdictions and professions argued strongly that this was a central consideration for having a single assessment provider and that the credibility of the exam in the eyes of qualification users (candidates, employers, the public at large) could be affected by having multiple providers competing for business – irrespective of whether this perception was based in reality or not.

With the exception of additional providers explicitly entering the market for non-profit reasons (and it is not clear that such an organisation exists), the current QLTS market is not sufficiently large enough in the short-term to sustain multiple assessment providers without the resource for assessment being reduced significantly (at risk to the confidence provided by the scheme). Over time, this dynamic could change if the market grows. It is unlikely that the pace of growth here will be so fast that it cannot be managed by the SRA and its single assessment provider. However, it is worth noting because the practicalities of meeting additional demand appears to be a key issue that legal regulators in other jurisdictions are having to address (usually in relation to the regulator's own in-house capacity to process applications, but also in terms of logistical issues such as the provision of sufficient exam space).

Having a single assessment provider may induce a dependency on the part of the SRA on a particular provider and associated risks as consequence (e.g. that there may be insufficient independence and a difficulty for the regulator to hold the assessment provider to account). This may just be a case of ensuring that effective safeguards are in place – and it is also worth noting that there are potential benefits from having this kind of working relationship. The real risk is in relation to the current assessment provider withdrawing suddenly from the market. This, though, is not in itself a decisive argument for having multiple assessment providers – instead, it suggests that the SRA should do what it can in the first instance to reduce this risk.

There may be in the long-term the potential for collaborative models of running assessments that mitigate the assumed costs / risks of moving to a multiple assessment provider model (for example, having a community of providers working jointly to offer a single set of assessments). There was little tangible appetite for this currently across the education and training community, but this, as much as anything, may reflect internal concerns that the until there is more evidence about the future scale of the QLTS market (under its current assessment format), there is no great push for involvement in assessment.

Training market

Separating the provision of assessment and training

The separation of the assessment and training role is perhaps even less contentious than the question of having a single or multiple assessment providers. There are numerous examples of similar models in other legal jurisdictions and other professions. Each of these cases shares general characteristics with the QLTS approach and experience to date in that there is a general principle that the assessment can be undertaken on a self-study basis, but with an acknowledgement that candidates may choose to benefit from additional training or support that may be offered commercially. All of the regulators interviewed for the study felt that it was crucially important not to be offering training and assessment. It was noteworthy and perhaps a little surprising that regulators interviewed for the study tend not to keep a close eye on this additional training provision. This is in part because much of the provision is ad hoc and/or typically offered by only a small number of providers.

The focus for some of these regulators is on ensuring that information is readily available for training providers (e.g. publishing past examination questions and examiners' comments or reports or model answers). While most regulators do not get involved in the training market on the basis of this separation between assessment and training, there is still an interest in it. There are examples of regulators who informally engage with providers to ensure that candidates are receiving accurate information (e.g. attending training sessions in an observer capacity).

The QLTS training market

One of the key concerns in relation to the QLTS training market has been whether there is a market failure preventing the emergence of an effective training market – and, if so, whether this should be a concern for the SRA. It is, though, increasingly apparent that, while still maturing, there is something approaching a critical mass of training provision for the QLTS. By September 2012, there are expected to be three providers active in the market. This includes a new entrant (QLTS School) and two previous providers of QLTT training (CLT and BPP). This is commensurate with the equivalent market for much more mature schemes in other jurisdictions and professions. Providers involved in the market based their decision to engage on assumptions about likely future size of the training market. It is notable that the three current providers provide quite different approaches. This may reflect the way that the training offer is evolving, but it means that candidates have access to support in different ways.

Even those other training providers that are not active in the QLTS training market reported that this was a decision they are keeping under review, and simply reflected perceived business risks related to the current market. Overall, there is no evidence of a current market failure in the provision of QLTS training, even though there remains understandable uncertainty about the nature of the future market.

Reflecting the SRA's options for intervention in the training market

Among the options set out for the SRA to potentially intervene in the training market, the idea to introduce a training requirement is easily discounted (it is disproportionate, a potential barrier to entry and unlikely to increase public protection). A second proposed option was to introduce a quality assurance dimension to the provision of training. This would not stimulate the training market (it may provide an additional barrier to training providers entering the market). There are, though, options for helping candidates to navigate the training market that do not necessarily involve the quality assurance or accreditation of training providers (e.g. the SRA could provide a list of training providers on its website). The third option proposed was the SRA providing either additional information to candidates or in providing additional support to training providers. Some activity has already been undertaken in this regard – including offering information/training days to potential providers and the provision of sample exam questions, and this has reportedly made a significant difference to providers (especially the development of more detailed assessment standards).

Conclusions and recommendations

The new assessments developed to underpin the QLTS scheme are widely-recognised to be robust, appropriate and innovative. The focus in the next phase should be on refinement, consolidation and dissemination. QLTS is strongly-aligned to the SRA's regulatory ambitions. Little compelling evidence has been presented that would suggest that removing or radically altering the current assessments would offer more or the same level of confidence. The transition to QLTS has had an impact on various stakeholders (potential candidates; law firms; training providers) that creates new challenges – but the existence of these challenges appear, in the main, to be not entirely relevant to the objectives of the scheme. The standout risk in relation to the current model relates to the contract with and dependence on the assessment provider. How these risks are mitigated going forward is likely to be central to the future success of QLTS.

The evidence is fairly clear in supporting the approach of having a single assessment provider for this type of scheme. It is the approach widely followed in most regulatory contexts. The key apparent distinction is less about the number of assessment providers, but whether the primary examination function is undertaken in-house by the regulator or externally (as under QLTS). While there is a high impact risk related to having a single assessment provider, which relates to what happens if the provider were to withdraw suddenly from the market, the likelihood of this risk occurring may be low and it is certainly a risk that can be managed and monitored by the SRA. Greater protection would be afforded through the SRA taking direct responsibility for setting and quality assuring examinations. However, it is also recognised that this is unlikely to be a realistic ambition in the short-term given the complexity of the design of the assessment. With some exceptions, the assessments undertaken inhouse by regulators tend to be simpler by design.

It is difficult to answer categorically what the impact on competence would be if assessment and training was provided by the same provider. There are relatively few meaningful comparators in other regulatory contexts. It is certainly difficult to argue that having training and assessment provided by the same provider offers the potential for a higher level of competence at admission than the current model. Alternatively, it is possible to argue that if the training market is considered to be particularly dysfunctional, then there would be an impact on solicitors' ability to qualify. This is a rather separate point – and it should be stated that there is no clear evidence that any such dysfunction in the QLTS training market exists (quite the opposite). There is a much more important point here, though, about perceptions of fairness and the integrity of the system in the eyes of QLTS 'users' (primarily candidates, but also the wider legal sector and, ultimately, the general public). There is a risk, in terms of perceptions, that an assessment provider offering revision courses could gain commercially from failing candidates. Even if this is unlikely to occur given sufficient exam controls, there is no way to stop candidates perceiving a potential conflict of interest in relation to their own experience of the QLTS process.

Recommendations to the SRA:

1. Maintain the current single assessment provider model in the short- to medium-term (for the next three years at least).
2. If possible, continue with the existing assessment provider relationship in the next phase, but, in doing so, consider setting an explicit timescale for re-tendering the contract on a competitive basis.
3. As part of the contract re-negotiation with the assessment provider, consider ways in which the risk the SRA of provider withdrawal can be minimised. Some of these options may already be included in the current contract, but in the next phase are likely to include penalty clauses for withdrawal and the possible transfer of some intellectual property for the QLTS to the SRA in conjunction with a future funding approach that ensures that the contract is commercially viable for the assessment provider based on current candidate volumes (which may be assumed to be something of a worst case scenario).
4. Provide additional help to candidates looking to navigate the QLTS training market through, as a minimum, the publication of a list of training providers offering QLTS training (couched as other regulators do by being clear that inclusion on the list is not an endorsement). Provide informal control over this list by ensuring that only providers engaging with the SRA (i.e. attending information sessions) are included on the list.
5. Continue to develop a relationship with training providers offering QLTS without regulating that market. Continuing to offer regular information sessions should be a core part of this activity, but the SRA should also consider offering to attend training sessions in an observer capacity.
6. In light of Recommendation #3, the SRA should consider whether it is possible to make more information about the QLTS assessments publicly-available. As a minimum, there should be a presumption of openness about the exams as seen in other professions and jurisdictions. While it is recognised that publishing multiple choice exams is problematic, a wider range of sample questions may possibly be made available.