Method

STAGE 1: Planning (Oct—Dec 2016)

1. The JA Team (JAT) comprised practitioners from the Governance and Justice Group (GJG); the Bluhm Legal Clinic at Northwestern University (Chicago); and Justice Mapping. The team worked closely with a dedicated project unit in the German Development Cooperation (GIZ). See Justice Audit Team below.
2. The JAT reviewed and updated the library of documents (see Library).
3. In close consultation with the GIZ project unit, the JAT met with heads of the justice institutions to secure the consent of each institution to sharing their data for the purposes of the Justice Audit; to agree the cut-off date of the Justice Audit as 31 December 2016; and to nominate institutional focal persons (IFPs) as their representatives on a Justice Audit Forum (JAF).
4. The JAF was first convened in December 2016 to review and agree the Terms of Reference of the JAT as well as the time-bound work plan. Thereafter the JAF met on four separate occasions in the course of the project under the chairmanship of Mustafizur Rahman, National Project Director, JRCP Project (GIZ).
5. The JAF comprised senior representatives from the following institutions:
   • Supreme Court
   • Police
   • Dept of Narcotics Control
   • Solicitor Wing (prosecution)
   • National Legal Aid Services Organisation
   • Bar Council
   • Department for Social Services (Child Detention Centres and Probation)
   • Prisons Directorate
   • Bangladesh Bureau of Statistics
   • Ministry of Law, Justice and Parliamentary Affairs
   • Ministry of Home Affairs
   • Department of Local Government
   • Ministry of Women and Children’s Affairs and Department of Women Affairs
6. The JAF steered the Justice Audit from the beginning: facilitating district visits and development of instruments (ensuring questionnaires and requests for supplementary data were relevant and appropriately worded), supporting the cleaning of data and ground truthing data findings.

STAGE 2: FRAMING (Jan-March 2017)

Data integrity review

7. The JAT visited Chittagong and Madaripur in January 2017 to visit each justice institution (police, prosecution, district legal aid offices, NGOs, courts, probation and prisons) to understand the process by which data was collected, stored and communicated up to the institutional centre / headquarters. The districts were identified by GIZ and approved by the NPD. Letters were dispatched in advance by the NPD and institutional focal points (IFPs) personally contacted their counterparts in the districts and agreed the schedule in each district with GIZ.
8. The full team (GIZ and GJG) met prior to departure to standardize the approach, questionnaires and agree the information sought. The teams debriefed individually at the close of each day and cross-checked key points and clarified queries between Chittagong and Madaripur. The full team then met in Dhaka to debrief.
9. The findings were that most institutions had a robust system for collecting, storing and communicating data up the chain on a monthly basis. The courts with their multiple registers were able to track individual cases through the system (from register to register) but had to count manually the volume of cases passing through. This inevitably led to discrepancies. Each adjournment was noted on a separate paper attached to each case file (usually three entries per side of A4 paper). One case filed eight years previously had over 100 adjournments. It was apparent that counting the adjournments per case was not feasible under the present system of data collection. While the office of the Public Prosecutor neither had the support staff nor was under any duty to capture data and report up to the Solicitor Wing in the MoLJPA.

STAGE 3: COLLECTING (Apr-Aug 2017)

10. The first data set collected was the national data for 2016 published by the respective institutions. The JAT then visited six districts (Gopalgonj, Comilla, Rangur, Khulna, Sylhet and Rajshahi) to compare nationally reported data with their sources.
11. Each visit followed a similar pattern. The visits began with a meeting comprising key justice stakeholders in the district to launch the visit and brief stakeholders on the Justice Audit. A schedule of meetings were then held with:
    • the metropolitan police, district police station, two police thanas;
    • the prison
    • the metropolitan and district courts, the judicial, metropolitan and executive magistrates (mobile courts);
    • village courts;
    • public prosecutor, assistant public prosecutors;
    • District Legal Aid Committee panel lawyers, NGO panel lawyers, paralegals and Restorative Justice Facilitators;
    • Probation; and
    • One stop Crisis Centre (OCC).
12. The purpose behind each visit was the same: validation of national data against source documents entered on forms agreed with the stakeholders and logging of offences in police thanas registered as ‘Other’.

13. The findings of the three teams visiting the six districts were strikingly similar:
    • when compared with the published national data (for the district), the court case data contained significant mismatches; and
    • the police definition of ‘Other’ (as in ‘Other’ offences which constitute the largest category of crime type, followed by Drugs) differed from district to district.

    The JAT never managed to clarify this definition.

14. Practitioner Surveys: the Justice Audit elicited the views of practitioners working in the criminal justice system (from police officers to assistant public prosecutors and lawyers; from magistrates and judges to their clerks; as well as jailors and probation officers) to raise their voices, identify major stresses and challenges affecting them in their work; and articulate their key frustrations with the criminal justice system. Questionnaires were drafted earlier and field tested during the six district visits.
15. The review of the national data sets disclosed a number of gaps. Supplementary Data Collection (SDC) forms were developed in consultation with the IFPs to be sent out to each institution in each district. These SDC forms disaggregated the national case data (by gender and age but also by case type / disposal etc) and collected further data concerning infrastructure, resources and governance (oversight and complaints mechanisms).
16. The JAT presented the package of SDC forms and Practitioner surveys at a second meeting of the Justice Audit Forum (JAF 2) which agreed to the distribution of SDCs and the revised Practitioner Surveys to the 64 districts. The JAF agreed to collecting the data by the end July 2017. As Ramadan followed by Eid-a-Fitr were approaching in May/June, this timeline proved unduly optimistic.

STAGE 4: INTERROGATING (Sep-Dec 2017)

Practitioner Surveys

17. The completed questionnaires were returned in hard copy to their headquarters in Dhaka and collected by the JAT. Some districts had over 100 responses while others had one, two or none. The JAT reviewed the responses by division to ensure a representation of views from the districts in each of the eight geographical divisions of Bangladesh.
18. The returned questionnaires identified the district of the respondent, his/her office (ie Judge or Magistrate) and gender (ie male of female). In entering the data, all personally identifying information was removed, including the name of the districts and remaining only with the type of office holder (ie judge, magistrate, clerk) and his / her gender.
19. The total number received is set out in the following table:
    

    Practitioner Survey by Institution and occupation	Number received
    Judges	286
    Magistrates	418
    Clerks (Peshkar / Administrative Officer)	894
    Police	516
    Prisons	115
    Public Prosecutors	132
    Probation officers	71

Supplementary Data Collection forms

20. The supplementary data forms were logged as they came in. Those in soft copy were entered in the institutional folder. Those in hard copy were entered into excel. A GJG team member who did not enter the data, spot-checked a minimum of six data entries by institution in six districts by comparing the hard copy with the excel entry. Once the spot checks were completed, these too were logged.
21. The Supreme Court research unit had designed a dash board for judicial officers in the districts to complete while the Police established a dedicated unit within police headquarters to receive and check the data.
22. The data were then reviewed: gaps in data, outliers, mismatched data (eg sentencing disposals compared to total convicted) and errors in totals were noted and forwarded to the IFP. The responses were entered. However, gaps, mismatches and outliers remained. The JAT sought – and obtained – permission from the IFPs to call the district officer concerned directly to clarify the data query.  Where the reliability of the data remained in question, data notes were entered alongside the data in the baseline.
23. The third meeting of the JAF took place (JAF 3) to review progress. Data cleaning took several months and was not completed until January 2018. Even then two institutions were awaiting data from several districts. By the end of January the complete baseline data sets by institution and for all 64 districts and metropolitan areas were submitted through GIZ to the relevant IFP for validation.

STAGE 5: DISCOVERING (Jan-May 2018)

24. A Letter of Agreement was signed by the MoLJPA, GIZ and BBS which authorised the BBS to conduct a nationally representative Citizen Household Survey and purposive Court User Survey. The BBS while experienced in conducting surveys had had no previous experience in conducting surveys about justice and especially not in surveying court users. The GJG’s director of research was then invited by the BBS to submit Methodology notes for both surveys which were approved by the Technical Committee of the BBS.
25. In addition, the GJG sub-contracted a separate firm of enumerators to support the BBS in the preparatory work for the surveys. The firm conducted eight cognitive interviews on the Court User Survey and 12 cognitive interviews on the Citizen Survey, according to pre-agreed selection criteria. They then followed with field tests of both surveys (48 Court User Surveys and 50 Citizen Surveys). After each round of testing, the questionnaires were amended in both English and Bangla translations – and back translated.
26. The field reports from the sub-contractor were used to supplement the Training Manuals prepared by the GJG for the BBS enumerators for each survey. The sub-contractor and GJG Director for Research then attended the two day Training of Master Trainers in Dhaka. During this training, the questionnaires were finally ‘tweaked’ and incorporated in a final set of questionnaires in Bangla and English translations and incorporated in the final training manuals (also in both Bangla and English translations). The Master trainers were then deployed to the eight divisions in the country to train the BBS enumerators, supported by GJG’s sub-contractor.
27. The BBS enumerators then conducted the surveys and entered all responses on to tablets. The BBS set out its own methodology as follows:

    The Citizen Household Survey data collection covered 6,400 households. 3,200 males and 3,200 females were interviewed, which were selected through the random sampling guided by the Technical Committee. The sampled households were attributed to all 64 districts (Annex-III: Distribution of Samples by Sex and District).The Court User Survey covered 1,428 interviews from all 64 districts as suggested by the technical committee. The interviewee was selected on purposive basis (proportionate non-probability sampling). The survey covered different courts also as suggested by the GIZ and GJG team to ensure the participation from lower courts in Bangladesh. The distribution of the number of interviews are below:

    Type of Court	Male	Female	Total
    Chief Judicial Magistrate Court	460	207	667
    Chief Metropolitan Magistrate Court	56	25	81
    District Session Judge Court	326	129	455
    Metropolitan Session Court	24	12	36
    Women & child Court	67	123	190
    Total	933	496	1,429

    Cleaned SPSS data set files with data dictionary, and frequency tables and key findings incorporating cross tabulations with key demographics for case analysis were provided to GIZ and GJG.

    The technical methodology report approved by technical committee was shared with the all technical committee members including LJD, GIZ, and GJG.

    As the data collection was conducted through CAPI, the dashboard link was provided to GJG and GIZ to see the progress on daily basis. The dashboard was also provided the facilities to see the location of data collection in the google map, the microlevel data. The response rate was 100% as the unavailable respondents were replaced with alternative household selected centrally.

    The data collection system was well designed with the monitoring system with both traditional and with the help of technology (CAPI dashboard). Interviews found with short duration was deleted and re-interviewed with close supervision to make sure that data quality is properly maintained.

    The GPS location tracking failed in a few remote areas as the mobile network coverage was not available. A different CAPI module was provided to re-capture the GPS location of the unchanged GPS locations if found. It could minimize the error of GPS locations.

    The mobile number of the respondents were collected though the questionnaire (which has not been shared). From the central control room phone calls were done to check the validity of the interviews randomly. If any negligence is identified the interviews form the particular area was deleted and re-interviewed with possible replacement samples (randomly selected).

    Officials from Statistics and Informatics Division including the Secretary, Additional Secretaries and high officials; Officials from BBS headquarters including the Director General, Deputy Director General, Directors, Joint Directors and other senior officials randomly visited the field to monitor the quality of data collection. The district and division level officers were closely involved to supervise the data collection.

28. The fourth JAF meeting took place (JAF 4) outside Dhaka over three days to review and critique a presentation of the Justice Audit. The JAF were shown a ‘model’ district. The JAF checked the nomenclature, commented on the clarity of the visuals employed and approved the model for replication in the remaining 63 districts, subject to the amendments being incorporated. The JAF also reviewed and commented on early national level views. In addition, the JAF interrogated the data findings suggested and strategies advanced for filtering cases out of the criminal justice system. The JAF’s consolidated comments were collected by GIZ at the end of May and communicated to the design team.

STAGE 6: DESIGNING (Jun-Jul 2018)

29. In June-July, Justice Mapping designed the web application for a launch in late July. Narratives in support of the visualised data were drafted in English and translated by GIZ into Bangla.

STAGE 7: VALIDATING (Jul-Sep 2018)

30. The Justice Audit was presented to the Minister of Law, Justice and Parliamentary Affairs in the presence of the Chief Guest, the Chief Justice of Bangladesh and the Minister of Home Affairs on 29 July 2018.
31. In August-September, the D3.js programming was completed along with database design and back end development. The system was fully tested before the Justice Audit web application was formally handed over to the MoLJPA.

THE JUSTICE AUDIT TEAM

The Justice Audit Team is a collaboration among a number of organizations and people. It is led by The Governance & Justice Group (GJG), which is comprised of: Justice (retired) Johann Kriegler, Dr Hania Farhan, David Morgan, Heather Goldsmith, Marcus Baltzer, Maliha Hussein, Kathryn English, Adam Stapleton, Radha Nair, Farah Alam, Hasanul Musaddek, Fazlul Chowdhury and Kashfiya Nawrin and was further supported by Henrietta Champion de Crespigny. The Bluhm Legal Clinic team was led by Dr. Tom Geraghty, Director, with students: Farzeen Tariq, Sarah Aagard, Samanta Suheen, Natasha Karamally and Ari Caldwell.

Justice Mapping is led by Eric Cadora with Charles Swartz and their technical team: Bill Shander, Andrey Yelbaev, Tatiana Temple, Alexandra Dubow, and Victor Radovinskiy.

The GIZ unit was led by Munir Uddin Shamim with Morshed Alam, Mahbubur Rahman Nazmi and Rasel Mehedi Parves with Kaiser Ul Islam Khan and Mohammad Shihab Uddin under the overall management of Promita Sengupta.