Skip to content

Contact sales

By filling out this form and clicking submit, you acknowledge our privacy policy.

Working with Formatted Text Files in R

May 4, 2020 • 8 Minute Read

Introduction

In this guide you will learn about the facilities R provides to work with formatted text files. Working with files is a very common task, especially in R. Most of the time data scientists have huge amounts of data on network shares or hard disks. Understanding how to access them and process these files is crucial, because the large amount of data to be processed is coupled with long runtime of scripts. The more efficient you are handling files, the more time you can save with your optimized code. First we will clarify what is meant by "formatted text file," then work with the interfaces provided in R.

Common Text Files

These formats will be fairly familiar to you:

  1. TXT
  2. CSV
  3. JSON (JavaScript Object Notation)
  4. XML (Extensible Markup Language)

These types are the most common ones for storing unstructured or structured data. When you are using a common file format with unstructured data you need to make sense of it. This means you need to understand each piece of information in those files and adjust your app accordingly. The situation is much easier when the data is structured. Unstructured data and regular-expression walk hand in hand; the regular-expressions allow you to parse out meaningful information while keeping the resource consumption relatively low. For more information on regular-expressions, check out this resource.

Prerequisite

The default R installation has no packages supporting your activity, but its versatile package repository allows you to add this functionality. You need to grab and install the readtext package. After firing up the R console, issue the following command:

      install.packages("readtext")

This way, the latest stable version is installed on your system. If you like to experiment with the newest version and its functionality you can install it from Github. The following commands will do that for you:

      install.packages("devtools")
devtools::install_github("quanteda/readtext")

To be able to install bleeding edge packages you need the devtools package installed, then the syntax below that line is the real deal. What happens here is that the following prefix is added to https://github.com/, and if you insert the URL https://github.com/quanteda/readtext in your browser it will take you to the source files for the package.

Action

In this section you are going to use the data available from the US about stolen guns. It is enough to download some CSV files and place them into the same folder. Spin up the R console and load the readtext library.

      library(readtext)

Right now you need to set the DATA_DIR variable, which is going to be your workplace. When you install the readtext package, it comes with some examples that are installed at the package's location.

      DATA_DIR <- system.file("extdata/", package = "readtext")

On a Windows machine you should see a similar output:

      1] "C:/Program Files/R/R-3.6.3/library/readtext/extdata"

It has several subfolders like the following:

      ├───csv
├───json
├───pdf
│   └───UDHR
├───tsv
├───txt
│   ├───EU_manifestos
│   ├───movie_reviews
│   │   ├───neg
│   │   └───pos
│   └───UDHR
└───word

If you want to load the data from the word folder, the following needs to be done:

      word_data <- readtext(paste0(DATA_DIR, "/word/*"))

Now word_data contains the following information:

      readtext object consisting of 6 documents and 0 docvars.
# Description: df[,2] [6 x 2]
  doc_id                                 text                 
  <chr>                                  <chr>                
1 21Parti_Socialiste_SUMMARY_2004.doc    "\"[pic]\r\nRés\"..."
2 21vivant2004.doc                       "\"http://www\"..."  
3 21VLD2004.doc                          "\"http://www\"..."  
4 32_socialisti_democratici_italiani.doc "\"DIVENTARE \"..."  
5 UK_2015_EccentricParty.docx            "\"The Eccent\"..."  
6 UK_2015_LoonyParty.docx                "\"The Offici\"..."

Now let's get back to the stolenguns folder. You can use the above option to simply specify the DIR_PATH folder for stolenguns, or use each file separately.

      > gun_data_q1 <- readtext("C:/Users/dszabo/Desktop/stolenguns/2016-first-quarter-stolen-guns.csv")
> gun_data_q2 <- readtext("C:/Users/dszabo/Desktop/stolenguns/2016-second-quarter-stolen-guns.csv")
> gun_data_q3 <- readtext("C:/Users/dszabo/Desktop/stolenguns/2016-third-quarter-stolen-guns.csv")
> gun_data_q4 <- readtext("C:/Users/dszabo/Desktop/stolenguns/2016-fourth-quarter-stolen-guns.csv")

Each variable will hold something similar.

      readtext object consisting of 7 documents and 9 docvars.
# Description: df[,11] [7 x 11]
  doc_id           text      Date   Brand   Model   Color Stolen   Stolen.From Status    Incident.number Agency
  <chr>            <chr>     <chr>  <chr>   <chr>   <chr> <chr>    <chr>       <chr>     <chr>           <chr> 
1 2016-first-quar~ "\"P1382~ 01/06~ HI POI~ "9MM"   "BLK" Stolen ~ Vehicle     Recovere~ B16-00694       BPD   
2 2016-first-quar~ "\"P1417~ 01/15~ JENNIN~ ""      "COM" Stolen ~ Residence   Not Reco~ B16-01892       BPD   
3 2016-first-quar~ "\"P1437~ 01/24~ CENTUR~ "M92"   ""    Stolen ~ Residence   Recovere~ B16-03125       BPD   
4 2016-first-quar~ "\"P1470~ 02/08~ TAURUS  "PT740~ ""    Stolen ~ Residence   Not Reco~ B16-05095       BPD   
5 2016-first-quar~ "\"P1504~ 02/23~ HIGHPO~ "CARBI~ ""    Stolen ~ Residence   Recovere~ B16-06990       BPD   
6 2016-first-quar~ "\"P1504~ 02/23~ RUGAR   ""      ""    Stolen ~ Residence   Recovere~ B16-06990       BPD   
# ... with 1 more row

There is an option where you can customize what is loaded into your data frame called document level metadata. You can take docvars from filenames, and it even allows you to name them individually. The devsep argument defines a separator or a regular-expression character string.

      gun_data_q4 <- readtext("C:/Users/dszabo/Desktop/stolenguns/2016-fourth-quarter-stolen-guns.csv",docvarsfrom = "filenames", dvsep = "_", encoding = "ISO-8859-1")

This should produce the following result.

      doc_id                                text            Date       Brand            Model     Color    Stolen         Stolen.From Status        Incident.number Agency docvar1                        
  <chr>                                 <chr>           <chr>      <chr>            <chr>     <chr>    <chr>          <chr>       <chr>         <chr>           <chr>  <chr>                          
1 2016-fourth-quarter-stolen-guns.csv.1 "\"P22093\"..." 10/25/2016 SMITH AND WESSON "SD9VE"   ""       Stolen Locally Vehicle     Not Recovered B16-42866       BPD    2016-fourth-quarter-stolen-guns
2 2016-fourth-quarter-stolen-guns.csv.2 "\"P22183\"..." 10/27/2016 TAURUS           "PT111G2" "BLACK"  Stolen Locally Residence   Not Recovered B16-43134       BPD    2016-fourth-quarter-stolen-guns
3 2016-fourth-quarter-stolen-guns.csv.3 "\"P22497\"..." 11/07/2016 SIG SAUER        "P290"    ""       Stolen Locally Vehicle     Not Recovered B16-44838       BPD    2016-fourth-quarter-stolen-guns
4 2016-fourth-quarter-stolen-guns.csv.4 "\"P22910\"..." 11/18/2016 TAURUS           "85UL"    "SILVER" Stolen Locally Residence   Not Recovered B16-46503       BPD    2016-fourth-quarter-stolen-guns
5 2016-fourth-quarter-stolen-guns.csv.5 "\"P23536\"..." 12/07/2016 SMITH & WESSON   ""        ""       Stolen Locally Vehicle     Not Recovered B16-48692       BPD    2016-fourth-quarter-stolen-guns
6 2016-fourth-quarter-stolen-guns.csv.6 "\"P23657\"..." 12/09/2016 COBRA            ".380"    "BLACK"  Stolen Locally Residence   Not Recovered B16-49060       BPD    2016-fourth-quarter-stolen-guns

The way you approach using the readtext module is very dependent on the actual formatting your data takes.

Conclusion

In this guide we have seen what facilities are provided by R to work with common formatted text files. We have seen what prerequisites are there to help us on our journey, and grasped the foundation that helps us move further. I hope this guide has been informative to you and I would like to thank you for reading it.