Next: , Previous: , Up: k9: Manual  

11 I/O and Interface

This chapter covers reading and writing to file (File I/O), interprocess communication (IPC), and working with pre-built shared libraries (csv, json, lz4, and zstd).

 0: r/w line
 1: r/w char
*2: r/w data
*3: k-ipc set    
*4: https get
 5: ffi/import

 `csv?`csv t;`json?`json t                                                   
 `lz4?`lz4 t;`zstd?`zstd t  

11.1 File I/O

k9 reads and writes to files in three different formats including line (list of strings), char (list of characters), and data (k9 format).

11.1.1 Example File I/O

Let’s begin with a simple example to creat a sample table, write to csv and then read it back in again. We’ll use a function so the table can easily grow to a larger size.

 g:{+`s`f`i!(x?"abc";x?10.;x?10)} / generate function for table

g[5]                              / generate table with 5 rows
s f          i
- ---------- -
c 1.296544   4
c 0.03771765 7
c 3.371475   0
b 1.352739   4
b 1.187619   5

`csv@g[5]                         / convert table to csv string

 "sample.csv"1:`csv@g[5]          / write to sample.csv file

 1:"sample.csv"                   / read from file

 `csv?1:"sample.csv"              / read from file and convert from csv
s f           i
- ----------- -
a 3.669518    1
a 0.001226037 7
b 2.792163    3
c 8.539221    7
b 9.333188    7

Given the small size of the file all this happens too quickly to notice the speed. Let’s give it a go with a bigger file.

 \t "sample.csv"1:`csv@g[_1e7]
 \t t:`csv?1:"sample.csv"

Ten million rows via csv in under 1/2 a second. Of course results will depend on hardware and number of columns also.

11.1.2 read line ⇒ 0:x

Read from file x.


11.1.3 write line ⇒ x 0:y

Output to string x (file name or null for stdout) the list of strings in y. y must be a list of strings. If y is a single string then convert to list via list.

 ""0:("blue";"red")      / "" represents stdout

 "file.txt" 0: ("blue"; "red") / write to file, one line per element

11.1.4 read char ⇒ 1:x

Read in file x (string).

 / file.bin contains the binary value 0x0010FA37

 / file.txt contains the ascii string 37\ntest\n

11.1.5 write char ⇒ x 1:y

Write to file x (string) data y (list of characters).



 "some.csv" 1:`csv [[]a:1 2;b:3. 4.]

 1: "some.csv"

a b
- -
1 3
2 4

11.1.6 read data ⇒ 2: x

Enterprise Only

Load file, eg. csv or from a (x 2: y) save. For the latter, one can find a “save then load” example in the next section.

s    t        e p  z   
---- -------- - -- ----
AABL 09:30:00 D 11 4379
AABL 09:30:00 B 40 3950

 2:`r                / read from file
a          b          c          d          e         
---------- ---------- ---------- ---------- ----------
0.5366064  0.8250996  0.8978589  0.4895149  0.6811532 
0.1653467  0.05017282 0.4831432  0.4657975  0.4434603 
0.08842649 0.8885677  0.23108    0.3336785  0.6270692 

11.1.7 write data ⇒ x 2: y

Enterprise only

Save to file x non-atomic data y (e.g., lists, dictionaries, or tables).

This example saves 100 million 8-byte doubles to file. The session is then closed and a fresh session reads in the file. Both the write (420 ms) and compute statistics from the file have impressive speeds (146 ms) given the file size (800 MB).

 `r 2:r              / write to file 

Start new session.

 \t r:2:`r;select avg a,sum b, max c, min d, sum e from r

11.2 IPC

k9 has the ability to comunicate between separate k9 instances via interprocess communication (IPC). These processes can be on the same or separate machines. A user will start multiple instances specifying the port (via the command line argurment -p) that the k9 session will use, opening a handle to that port, and then running remote commands via the handle and 3: or 4:. The commands to be run are sent as a string, eg. “3+12”.

11.2.1 k-ipc ⇒ 3:x

Open a connection to port x and return a handle. If the port has been forwarded from another machine (eg. ssh -L then this handle allows remote execution.

 / before running this session start another k session in another
 / terminal window specifying that port 1251 should be used
 / k -p 1251
 h:3:1251 / create handle to port 1251 and save to h
 h        / h is an integer

11.2.2 set ⇒ x 3:y

Exectute string y using handle x. Handle x should have already been created using k-ipc. set is asynchronous, thus once the command is sent the session immediately returns to the user. One will not be able to return any results via set. If a result is required then one should use get.

 h:3:1251               / create handle to port 1251
 h 3:"a:12"             / set a to 12 on remote session
 h 3:"n:_1e8;b:1?n?1."  / set b to a random number on remote

11.2.3 https ⇒ 4:x


11.2.4 get ⇒ x 4:y

Execute string y using handle x. Handle x should have already been created using k-ipc. get is synchronous, thus once the command is sent the session waits until there result is ready.

 h:3:1251               / create handle to port 1251
 h 3:"a:12"             / set a to 12 on remote session
 h 3:"n:_1e8;b:1?n?1."  / set b to a random number on remote
 h 4:"a"

 h 4:"b"

 a:"local"              / set a locally to a string value


11.3 ffi/import

These topics will be covered in the Foreign Functions chapter.

11.3.1 export, import csv ⇒ ‘csv@x, `csv?x

Enterprise only

csv tools allows one to export tables into csv and import from csv. Generally these functions are used with file i/o 1:

 "test.csv"1:`csv@[[]x:`a`b;y:3 2]  / write sample table to file

 `csv?1:"test.csv"                  / read csv from file
x y
- -
a 3
b 2

11.3.2 export, import json ⇒ ‘json@x, `json?x

Export object x to json format or import string x from json format. The json shared library will be automatically pulled from the Shakti website into the current directory. Use of symbolic links with path on the binary may cause issues.

 "test.json"1:`json@[[]x:`a`b;y:3 2]


11.3.3 lz4 zstd ⇒ `lz4 `zstd

lz4 and zstd are compression algorythms which can be used to reduce the size of files. As with other foreign libraries k9 does the work behind the scenes. Here is an example of writing a large csv file to disk and reading it back in. In the second (lz4) and third (zstd) examples the csv’s are compressed and decompressed with lz4 and zstd respectively. One can look at the file sizes to see how well the compression worked.



\t tc:`csv?1:"t.csv"            / read from csv

"t.lz4"1:`lz4@`csv@g[_1e7]      / write to lz4

\t tz:`csv?`lz4?1:"t.lz4"       / read from lz4 compressed csv

 "t.zstd"1:`zstd@`csv@g[_1e7]   / write to zstd

 \t tz:`csv?`zstd?1:"t.zstd"    / read from zstd compressed csv

Next: Foreign Functions, Previous: Knit Functions, Up: k9: Manual