Physical Therapy Korea

The prudent usage and management of the clinical assessment tools is as important as effective intervention for the research related to stroke and brain injury rehabilitation [1]. Especially, in multi-center randomized clinical trials (RCTs), the failure of an RCT in the rehabilitation is often related to the selection of inappropriate rehabilitation outcome measures (ROMs) [2] or uncertainty in the consistency of scores across sites and time due to the lack of the operational details of test administration [3]. Additionally, therapists or researchers often make mistakes when filling out scoring sheets using a pen or pencil, converting analog data into digital data (i.e., Excel file), or merging scores from multi-sites into one data file. Therefore, the need for outcome management systems for the RCT has increased.

In this study, we have developed a web-based platform, the Korean Rehabilitation Outcome Measurement (KoROM) (https://korom.co.kr/), for managing ROMs measured by physical or occupational therapists at multiple sites. The KoROM is a specialized online platform for entering clinical assessment scores directly using a PC or tablet, educating therapists, and managing and/or downloading the ROMs intensively. Most hospital and clinics utilize their own electronic medical record (EMR) system, which shows all registered patients’ information (i.e., patient demographics, medical history, examination results, treatment outcomes, surgery records, hospitalizations, discharges, outpatient visits, and health screening records) [4,5]. However, EMRs are not specific to the ROMs and are restrictive in expanding the coverage of clinical assessments. In addition, patients do not have easy access to their medical information [6], and therapists who are not enrolled in the EMR system also do not have permission to see the EMR [7]. Therefore, the KoROM has been developed to overcome these limitations in managing ROMs.

Recently, online platforms or applications on smartphones, laptops, or tablet PCs have been used for measurements in the rehabilitation field. Individuals, with or without disease, can now easily measure clinical assessments such as the 32-sit-to-stand, the timed-up-and-go, and the 5-times-sit-to-stand using a mobile application [8], the simple range of motions in the knee or shoulder joints [9,10], or cervical or lumbar spines [11,12]. Additionally, telerehabilitation or virtual reality programs provide patients with the opportunity to receive therapies at home [13-16]. However, there are few programs that systematically manage clinical assessment scores, which patients can also access [6]. Moreover, current EMRs in South Korea do not provide information on which assessment tools exist for each level of International Classification of Functioning, Disability, and Health (ICF) model, proposed by the World Health Organization (WHO) to evaluate patients at multiple levels. Furthermore, there is a lack of web platforms that provide therapist training and standardized courses for reliable rehabilitation clinical assessment.

Therefore, the aim of this study was to develop an interactive web-based platform for patients with neurological disorder, especially for patients with stroke or brain damage, and physical or occupational therapists to enter and manage clinical assessment scores based on the ICF model. In addition, therapist education and standardization can be accomplished within the system we developed.

The development of the KoROM took place from June 2021 to July 2022. During this period, the process involved searching for clinical tools for the ROMs, designing and developing the web platform, and refining the platform based on feedback from experts in rehabilitation and computer science. This study was approved by the Institutional Review Board at Jeonju University (IRB no. jjIRB-211115-HR-2021-1122).

1. Collection and Analysis of Requirements

Through online searches using PubMed, Google, Google Scholar, and Google Play, we searched for “rehabilitation (or physical therapy) apps,” “databases for rehabilitation (or physical/occupational therapy),” “online evaluation process,” and “electronic medical record.” We referenced WebPT (https://www.webpt.com/) for entering scores and the education center in the Academy of Neurologic Physical Therapy (ANPT) for educative videos on assessment tools. The general procedure for developing the KoROM is shown in Figure 1.

Figure 1. Flowchart for the KoROM development. ROMs, rehabilitation outcome measures; ICF, International Classification of Functioning, Disability, and Health; EDGE, evidence database to guide effectiveness; ANPT, Academy of Neurologic Physical Therapy; KoROM, Korean Rehabilitation Outcome Measurement.

We thoroughly searched for the ROMs, especially in physical or occupational therapies, based on the following three resources.

Resource 1: Clinical assessment tools eligible for medical insurance claims were downloaded from the Health Insurance Review and Assessment Service (HIRA) in South Korea. Given their extensive use in clinical settings, we compiled clinical assessment tools recognized by medical insurance and applicable to patients with stroke and brain injuries.

Resource 2: Our team researched the most commonly used clinical assessment tools for individuals with neurologic disorders in South Korea via a survey from 201 therapists [17].

Resource 3: The ROMs with evidence levels 3 and 4 from the evidence database to guide effectiveness (EDGE) of ANPT were selectively chosen. Level 3 measures were recommended, with good psychometrics but may lack information about validity, reliability, or responsiveness. Level 4 measures were highly recommended with excellent psychometrics, valid and reliable, and some data on responsiveness. We believe that ROMs with at least level 3 might be suitable for better patient measurement.

We sorted all the ROMs found through the three resources above and selected the most overlapping assessments to be included in the KoROM.

According to the ‘ICF-A Practical Manual’ in WHO and the ‘Rehabilitation Measures Database’ at the Shirley Ryan Abilitylab (https://www.sralab.org/rehabilitation-measures), the ROMs selected above were categorized into one of the ICF levels (Body Function & Structure, Activity, or Participation). The KoROM was designed for therapists to select at least one ROM at each ICF level.

2. Design of System Architectures

The KoROM has been developed in three distinct versions: therapist, patient, and master. The primary functions of the therapist version include patient registration, clinical score entry, and management. Additionally, the KoROM is capable of teaching clinical assessment and downloading scores. Score review/interpretation and user management are the primary functions of both the patient and master versions of the KoROM, respectively. Security of personal information, including patient evaluation records, is paramount. Therefore, data collected by KoROM is not shared with unspecified individuals. To identify therapists and patients, only the minimum necessary information (name, date of birth, mobile phone number) is required during registration. In addition, when downloading data in Excel format (Microsoft Co.), personal information, excluding the patient’s name, is not included, ensuring that specific patients cannot be identified. More details related to the security issues are described in Table 1.

Table 1 . Features of three different versions of the KoROM.

	Therapist	Patient	Master
Features provided	Managing patient information Entering and managing scores for the ROMs Educating therapists by providing educational videos or standardization processes Downloading scores for the ROMs	Monitoring scores with short explanation Printing out the scores in PDF format	Managing patients and therapists’ information Downloading scores for the ROMs Checking the therapist's learning and standardization history of ROMs
Scope of security	Collected personal information Therapist information: register by entering only the minimum information required to identify the therapist (name, mobile phone number, date of birth). Patient information: enter only the minimum information required to identify the patient (name, mobile phone number, date of birth). Personal information and patient assessment information are not shared with other institutions. All personal information except the name will not be downloaded when downloading data. It is notified and agreed that the patient and the owner of KoROM own the entered data.

ROMs, rehabilitation outcome measures; KoROM, Korean Rehabilitation Outcome Measurement..

3. Pilot Testing for the System

A total of six experts in physical therapy or computer science participated in the pilot test of the KoROM. The group consisted of three physical therapists (S.C, Y.J, and J.Y), two researchers (Y.S and S.H), and one computer scientist (S.Y). They reviewed the beta version of the KoROM and provided feedbacks. All physical therapists and researchers were licensed physical therapists with at least 3 years of clinical experience. Additionally, except S.Y, all of them had experiences in stroke rehabilitation research.

All participants thoroughly explored the therapist and patient versions of the KoROM for 1 week. In the therapist version, they registered patients, entered scores, watched educational videos for clinical assessment, filled out self-checklists for therapist standardization, and downloaded evaluation scores in Excel. In the patient version, they checked how the scores entered by therapists were displayed and interpreted for patients. The participants did not necessarily enter the actual patient scores; rather, they needed to check the features provided in the KoROM by entering virtual scores. The results in the following section showed the final version of the KoROM, which was edited based on feedback from the experts.

1. Selected Rehabilitation Outcome Measures

For the clinical assessment of the upper extremity, the following tools were selected: Fugl-Meyer Assessment for Upper Extremity (FMA-UE), Wolf Motor Function Test, Box & Block, Stroke Impact Scale, Motor Activity Log, and Actual Amount of Use Test. For the lower extremity, the selected tools were: Fugl-Meyer Assessment for Lower Extremity (FMA-LE), Berg Balance Scale, Functional Gait Assessment, Activities-specific Balance Confidence Scale, 10-Meter Walk Test, 6-Minute Walk Test, and 5 Times Sit-to-Stand. Additionally, basic measurements such as manual muscle test (MMT), range of motion (ROM), modified Ashworth scale (MAS), and both dynamic and static sitting and standing balances were included in the KoROM.

2. System Architectures

The KoROM for therapists includes four main menus: patient management, assessment, assessment tool information, and data download (Figure 2A, 2B and Table 1). In the patient management menu, therapists can register new patients. If their patients are already registered in the KoROM, they can search for their patients and pair the therapist and patient for further assessments. Under the tab of ‘patient searching,’ the basic information such as diagnosis, disease onset date, lesion area, and evaluation dates can be viewed.

Figure 2. The KoROM. (A) Welcome page of the KoROM, (B) patient management screen in the therapist version. KoROM, Korean Rehabilitation Outcome Measurement.

In the assessment menu, therapists can choose the ROMs based on the ICF model (Figure 3A). When therapists select a specific clinical assessment, they can enter the original score on the KoROM platform (Figure 3B). For patient-reported outcomes, where the patient needs to provide answers, the patient can view the KoROM screen and enter their own scores. The KoROM provides the average scores and asks for final confirmation before saving the scores.

Figure 3. The therapist version of the KoROM. (A) The clinical ratings categorized according to the ICF model, (B) the screen for entering the scores. KoROM, Korean Rehabilitation Outcome Measurement; ICF, International Classification of Functioning, Disability, and Health; ROM, range of motion; MMT, manual muscle test; MAS, modified Ashworth scale.

In the assessment tool information menu, therapists can learn about each clinical assessment, including testing positions, evaluation methods, scoring, and interpretation of the results. The KoROM provides educational videos for most of the clinical assessments used in the system (Figure 4A). In addition, the self-checklist, a part of the therapist standardization process, is available in this menu to help therapists become familiar with the clinical assessment tools (Figure 4B).

Figure 4. The therapist version of the KoROM. (A) Therapist training videos, (B) self-checking screen to standardize therapists. KoROM, Korean Rehabilitation Outcome Measurement.

Lastly, in the data download menu, all scores entered in the KoROM platform can be downloaded in Excel. Score for each item or averaged scores for each assessment or patient can be downloaded. Additional options allow therapists to specify the test, patient, evaluation dates, enabling them to view and download the patients’ scores as needed.

In the patient version of the KoROM, patients can select upper and lower extremities to monitor their scores (Figure 5A and Table 1). Compared to the therapist version, the user interface is simpler and uses ordinary language for ease of use. Once patients click on the name of the assessment tools, they can see the scores and a brief summary of the results. Additionally, they can print out the summary chart to share with other therapists or clinicians if needed.

Figure 5. The patient and master versions of the KoROM. (A) The screen a patient sees to check their score in the patient version, (B) a table to check if therapists have viewed the videos. KoROM, Korean Rehabilitation Outcome Measurement.

In the master version, the information of all users (patients or therapists) can be viewed (Table 1). This includes basic user information, clinical assessment scores, and educational course management activities such as watching videos or completing the self-checklist (Figure 5B). Additionally, clinical scores can be downloaded from this menu.

In this study, we developed a web-based platform named KoROM for entering and managing the scores of outcome measures in rehabilitation. Therapists can learn and practice the ROMs on the KoROM, while patients can view their scores whenever they want.

The KoROM is somewhat like the current EMR system, but it has several unique but strong aspects to use in rehabilitation. First, patients can view their clinical achievements via the KoROM. Both therapists and patients can register patients, allowing for patient-therapist matching. Consequently, patients can view their scores and results uploaded by therapists in the patient version of the KoROM. Although ‘therapist-patient matching’ in the KoROM might seem more complicated for therapists than an EMR system, it significantly increases patient involvement in rehabilitation. If patients and their caregivers, including family members, are aware of the assessment scores, it will facilitate patient- and family-centered care [18-20], which is a crucial aspect of modern physical and occupational therapies. This awareness is also expected to motivate patients and enhance the effectiveness of their treatment [21]. Advanced countries like Austria, Canada, and Denmark have already launched personal health records (PHRs), allowing patients to access their health information via the patient portal. South Korea is also conducting a national-level PHR project [22], and the KoROM could serve as an experimental platform for PHRs.

Second, the KoROM can be utilized as an educational tool for therapists. The KoROM provides two methods for therapist education: instructional videos on how to correctly evaluate and score patients and a self-checklist as part of the therapist standardization process. Lee et al. [17] reported that approximately 20% of therapists in their study did not use clinical assessment based on the ICF model due to a lack of information or knowledge. Using the clinical assessment tool correctly takes practice. For example, the FMA-UE, which is used worldwide, has a relatively simple rating scale from 0 to 2, but training is recommended to achieve reliable scores [3,23]. Another study shows that a single lecture is insufficient for accurate evaluation; instead, at least three practice sessions with real patients improve evaluation quality [24]. Therefore, both educational videos and the self-checklist in the KoROM would be helpful for therapists not familiar with clinical assessments.

Another key feature of the KoROM is its requirement for clinical assessments based on the ICF model. Since 2001, WHO has recommended evaluating patients from multiple perspectives within the ICF model framework [25]. Therapists in clinics or hospitals often focus more on evaluations at the body function and structure level (e.g., ROM, MMT, or MAS) rather than at the activity & participation levels [26]. This focus may be due to patients’ medical status, where strength or ROM recovery is a primary goal in acute or subacute stages, or environmental circumstances that restrict instrumental activities of daily livings. However, patients about to be discharged or outpatients need to be evaluated at the activity or participation levels [27,28]. Furthermore, for evaluating mobility and self-care conditions in the pediatric rehabilitation area, the ICF model can be a valid tool after extracting ICF code [29]. Therefore, therapists need to evaluate their patients with the ICF model, making the KoROM a valuable practice tool.

Lastly, the KoROM can be expanded and specialized for physical and/or occupational therapies. The current EMR system provides comprehensive medical information, but therapists cannot share scores within the basic EMR [30] or upload the scores of clinical assessments not provided by the EMR system. The KoROM includes clinical assessment tools for rehabilitation that are reimbursable in Korea and widely used and well-researched tools with psychometric information worldwide. If therapists need to manage scores of new clinical assessment tools not in the KoROM, the developers can quickly add a ‘tab’ for the new assessment. Thus, the KoROM may be useful for multi-site RCTs or small clinics and developmental centers for children with disabilities. Recently, therapists in South Korea have been able to open their clinics in the form of developmental centers for children with developmental disorders or community-based exercise centers for the elderly. The KoROM can be potentially used in these small centers. In countries like the United States, where private clinics can be established, thorough management of ROMs is achieved using all-in-one services like WebPT or Empower EMR (from patient appointments to payments). In the future, the KoROM need to provide an integrated solution that covers not only evaluation and treatment details but also patient appointments and payments. Additionally, future effort is needed to automatically link KoROM data to the hospital EMR system.

A limitation of this study is that we did not test the patient version. We plan to conduct usability tests on the patient version as well as the therapist version. After these tests for real-world implementation, we will modify and release the final KoROM to public. Patient registration is also somewhat difficult compared to the existing system due to the patient-therapist matching part. This suggests the need for a therapist manual to reduce user difficulty. In addition, the KoROM is currently limited by its design to focus primarily on stroke and brain injury patients, but in the future, we plan to add assessment areas for musculoskeletal and cardiorespiratory injuries as well as children with disabilities. Finally, there is the issue of data security. Patients are assigned IDs, but users must register their personal information to retrieve it if they lose their username and password. Privacy needs to be closely managed.

In this study, we developed a web platform to manage scores for ROMs for people with stroke and brain injury. The platform includes score entry, education on the assessment tools, a self-checklist for therapist standardization, and a data download function. In addition, the platform emphasizes and expands the rehabilitation domain compared to existing EMR systems and allows patients to freely review therapist assessments. The KoROM can be used not only in multi-site RCTs, but also in community rehabilitation exercise centers.

In the preparation of this manuscript, generative AI and AI-assisted technologies, specifically ChatGPT developed by OpenAI, were used for proofreading and language refinement.

This work was supported by the 2022 Research-Year Grant of Jeonju University and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2021R1F1A1045506).

No potential conflicts of interest relevant to this article are reported.

Conceptualization: SK, JJ, HL. Data curation: SK. Formal analysis: SK. Funding acquisition: SK. Investigation: SK, JJ, HL. Methodology: SK, JJ, HL. Project administration: SK. Resources: SK. Software: SK, JJ, HL. Supervision: SK. Validation: SK. Visualization: SK, JJ, HL. Writing - original draft: SK. Writing - review & editing: SK, JJ, HL.