patb/tf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

-add copyright

Browse Source 
-shorten variable names (for size)
-simplify sed() command scan
-sed backup extension includes the . dot now
-add extended commands in tf_extend.py
This commit is contained in:
Pat Beirne
2021-04-06 07:26:24 -04:00
parent 34f88c5948
commit 83857b5571
3 changed files with 209 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
README.md
View File

@@ -2,6 +2,8 @@
A module for manipulating files in the *MicroPython* environment. A module for manipulating files in the *MicroPython* environment.
[TOC]
## Oveview ## Oveview
I discovered *MicroPython* when working on the ESP8266 processor. Everything seemed very nice, except it was awkward moving files around. All the methods I could find required a back-and-forth with the programmer's desktop. I discovered *MicroPython* when working on the ESP8266 processor. Everything seemed very nice, except it was awkward moving files around. All the methods I could find required a back-and-forth with the programmer's desktop.
@@ -71,6 +73,8 @@ tf.cp('log.txt','log.bak')`
Simply copies a source file to a destination file. Filenames may include folders or . or .. prefixes. The destination is overwritten if it exists. This function reads-&-writes one line at a time, so it can handle megabyte files. Typical speeds are 100kB/sec on an ESP8266. Simply copies a source file to a destination file. Filenames may include folders or . or .. prefixes. The destination is overwritten if it exists. This function reads-&-writes one line at a time, so it can handle megabyte files. Typical speeds are 100kB/sec on an ESP8266.
**NOTE** this function *only works on text files*. Line lengths of up to 4096 work fine on the ESP8266.
#### cat() #### cat()
``` ```
@@ -124,7 +128,7 @@ tf.grep('config.ini', '\[\w*\]', numbers = True)
#### sed() #### sed()
``` ```
sed(filename, pattern, bak_ext="bak") sed(filename, pattern, bak_ext=".bak")
in: filename the file to edit in: filename the file to edit
pattern a sed pattern, involving one of "aidsxX" pattern a sed pattern, involving one of "aidsxX"
bak_ext the extension to use when creating the file backup (without the dot) bak_ext the extension to use when creating the file backup (without the dot)
@@ -172,7 +176,9 @@ s/thier/their/
s@ratio\s*=\s*num/denom@ratio = num/denom if denom else 0@ s@ratio\s*=\s*num/denom@ratio = num/denom if denom else 0@
``` ```
**Note**: you will need some free space on your disk, the same size as the source file, as a backup file is *always* made. To edit an 800k file, you should have 800k of free space. **Note**: The function version of sed() can have embedded space characters in the pattern; the command line version (below) requires single-quotes around patterns that have space characters.
**Note**: You will need some free space on your disk, the same size as the source file, as a backup file is *always* made. To edit an 800k file, you should have 800k of free space.
**Note**: The functions for **Note**: The functions for
@@ -235,17 +241,18 @@ Commands with invalid syntax return a line of information, and are ignored. Non
In its present form, the module has these limitations: In its present form, the module has these limitations:
* filenames are limited to 255 chars * filenames are limited to 255 chars
* search patterns involving \ escapes may or may not work properly * files must be text
* the esp8266 implementation does not allow \1,\2 type pattern substitution * or have a `\n` at least every 4096 characters
* `sed()` requires lines <=2048 characters, and this `sed()` won't match binary chars
* search patterns involving \ escapes other than `\'` probably won't work
* in the simple shell * in the simple shell
* filenames must not have spaces * filenames must not have spaces
* patterns with spaces ***must*** be quoted * patterns with spaces ***must*** be quoted
* the target of `cp`and `mv` *cannot* be a simple a directory-name as in Linux; write the whole filename *w.r.t,* the current directory * the target of `cp`and `mv` *cannot* be a simple a directory-name as in Linux; write the whole filename *w.r.t,* the current directory
* the complexity of pattern matching is limited. * the complexity of pattern matching is limited.
* try to format the grep patterns so they avoid deep stack recursion. For example, '([^#]|\\#)\*' has a very generous search term as the first half, and can cause deep-stack recursion. The equivalent '(\\#|[^#]\*)' is more likely to succeed. * try to format the grep patterns so they avoid deep stack recursion. For example, '([^#]|\\#)\*' has a very generous search term as the first half, and can cause deep-stack recursion. The equivalent '(\\#|[^#]\*)' is more likely to succeed.
* with sed-search-and-replace, the parsed line *includes* the terminal \n, so if you replace the text all the way to the end of the line, you will delete the terminal \n and this line will merge with the subsequent line * with sed, lines are parsed and saved one-line-at-a-time, so pattern matching to \n and \r does not work
* if the replacement is '', the line will appear to vanish, e.g. `s/^#.*//` will delete comment lines * this simple shell is different than [mpfshell](https://github.com/wendlers/mpfshell) in that this shell runs entirely on the target device. There is no allowance in this shell for transferring files in/out of the target.
* pattern matching to \n and \r does not work
## Examples ## Examples
@@ -276,14 +283,16 @@ Search a log file for an incident
[command line] [command line]
grep [Ee]rror log.txt grep [Ee]rror log.txt
grep '2021-02-12 16:\d\d' log.txt grep '2021-02-12 16:\d\d' log.txt
[search and keep a record ] # search and keep a record
cp log.txt log.details cp log.txt log.details
sed 'x/2021-02-12 16:\d\d` log.details sed 'x/2021-02-12 16:\d\d` log.details
``` ```
## Installation ## Installation
Move the 'tf.py' file over to the target. You can use `webrepl` [command line program](https://github.com/micropython/webrepl) or the **WEBREPL** [web page](https://micropython.org/webrepl/) . ~If you need help with getting connected to your MicroPython board, there are excellent howto guides here and here~ TODO!
Move the 'tf.py' file over to the target. You can use `webrepl` [command line program](https://github.com/micropython/webrepl) or the **WEBREPL** [web page](http://micropython.org/webrepl/) . If you want the command line extensions, then send over the `tf_extend.py` file as well
Once the module is present in the file system of the target, you can use the **REPL** command line interface to invoke it Once the module is present in the file system of the target, you can use the **REPL** command line interface to invoke it
@@ -315,12 +324,30 @@ This is the *simple command line*. You can type `dir` to get an idea of what's a
/test_dir$ /test_dir$
``` ```
If you don't need the *simple command line*, you can still use the methods listed above. Feel free to cut the `tf.py` module in half by deleting everything below the line If you don't need the *simple command line*, you can still use the methods listed at the top of this readme. Feel free to cut the `tf.py` module in half by deleting everything below the line
``` ```
def help(): def ext_cmd(a):
``` ```
## Extensions
I found the simple command line so useful, I added some extra non-file-related functions. These are included in the optional filr `tf_extend.py'. the available command list is extended to include
```
scan # scan and show the local AP's
connect essid password # create a persistent wifi connection
ifconfig # show current ip address
host <domain.name> # do an DNS lookup
freq [160 | 80] # get/set the ESP8266 frequency
exec <python-filename> # execute a small python file
free # display the heap size: used + free
```
The `tf.py` module checks to see if the `tf_extend.py` files exists, and forwards unknown commands to it. The `help` system also extends when the extension file exists.
Installing the extensions module uses about 2k of flash/disk space and 2kB of heap ram.
## Performance ## Performance
Typical performance on an ESP8266 @80MHz, 90kB log file, 1200 lines; serial connected terminal @115200baud Typical performance on an ESP8266 @80MHz, 90kB log file, 1200 lines; serial connected terminal @115200baud

180
tf.py
View File

@@ -1,122 +1,128 @@
# tf Text File manipulations # tf Text File manipulations
# for micropython and other tiny environments # for micropython and other tiny environments
# (c) Pat Beirne patb@pbeirne.com
#NOTE: the ESP8266 port cannot to \1,\2 type replacements in the s/search/replace/ operator
import re,os,sys,gc import re,os,sys,gc
def _file_scan(src,dest,start=1,end=0xFFFFFFFF,numbers=False,grep_func=None): # ====legend====
#src is a filename, dst is an open handle # e=end
# f=input file
# g=output file
# i=iteration variable
# m=line range
# r=replace text
# s=start/search/string
# sp=search regex
def transfer(src,dest,first=1,last=0xFFFFFFFF,numbers=False,grep_func=None):
#src is a filename, dst is a handle
i=0 i=0
try: try:
with open(src) as f: with open(src) as f:
for line in f: for lin in f:
i=i+1 i=i+1
if i<start or i>end: if i<first or i>last:
continue continue
if grep_func and not grep_func(line): if grep_func and not grep_func(lin):
continue continue
if numbers: if numbers:
dest.write(str(i)+' ') dest.write(str(i)+' ')
dest.write(line) dest.write(lin)
except: except:
print("could not open file {}".format(src)) print("could not open file {}".format(src))
def cp(src_f, dst_f): def cp(src,dest):
try: try:
with open(dst_f,'w') as g: with open(dest,'w') as g:
_file_scan(src_f,g) transfer(src,g)
except: except:
print("could not write to file {}".format(dst_f)) print("could not write to file {}".format(dest))
def grep(filename, pattern, numbers=False): def grep(filename, pattern, numbers=False):
m=re.compile(pattern) m=re.compile(pattern)
if not m: if not m:
print("grep() called with invalid pattern") print("grep() called with invalid pattern")
return None return
_file_scan(filename,sys.stdout,numbers=numbers,grep_func=(lambda x:m.search(x))) transfer(filename,sys.stdout,numbers=numbers,grep_func=(lambda x:m.search(x)))
print()
def cat(filename, first=1, last=1000000, numbers=False, title=True): def cat(filename, first=1, last=1000000, numbers=False, title=True):
if title: if title:
print("===={}=====".format(filename)) print("===={}=====".format(filename))
_file_scan(filename,sys.stdout,first,last,numbers=numbers) transfer(filename,sys.stdout,first,last,numbers=numbers)
print()
def sed(filename, sed_cmd, bak_ext="bak"): def sed(filename, sed_cmd, bak_ext=".bak"):
#print("sed() called with sed_cmd=<{}>".format(sed_cmd))
# parse the sed_cmd # parse the sed_cmd
# group 1,3 are the n-start, n-end group 4 is command # group 1,3 are the n-start, n-end group 4 is command: aidsxX
g=re.search("^(\d*)([,-](\d+|\$))?\s*([sdaixX].*)",sed_cmd) a=re.search("^(\d*)([,-](\d+|\$))?\s*([sdaixX].*)",sed_cmd)
if not g: if not a:
print("sed() failed; 2nd argument must be a number followed by one of sdaixX; no changes applied") print("sed() failed; 2nd argument must be a number-range followed by one of sdaixX; no changes applied")
return 0,0 return
cmd=g.group(4) cmd=a.group(4)
#print("sed() cmd parsed into <{}>,<{}> and <{}>".format(g.group(1),g.group(3),g.group(4)))
start,end=(1,1000000) s,e=(1,1000000)
if g.group(1): if a.group(1):
start=end=int(g.group(1)) s=e=int(a.group(1))
if g.group(3): if a.group(3):
end=1000000 if g.group(3)=='$' else int(g.group(3)) e=1000000 if a.group(3)=='$' else int(a.group(3))
op=cmd[0] op=cmd[0]
if op not in "sdiaxX": if op not in "sdiaxX":
print("sed requires an operation, one of 's,d,i,a,x or X'") print("sed requires an operation, one of 's,d,i,a,x or X'")
return 0,0 return
#print("sed command parser of <{}> returned {} {} {} {}".format(cmd,sr,de,ins,add)) #print("sed command parser of <{}> returned {} {} {} {}".format(cmd,sr,de,ins,add))
if op in "sxX" and len(cmd)<2: if op in "sxX":
if len(cmd)<2:
print("invalid sed argument") print("invalid sed argument")
return (0,0) return
dl=cmd[1]
if op=='s': if op=='s':
dl=cmd[1]
gs=re.search("s"+dl+"([^"+dl+"]*)"+dl+"([^"+dl+"]*)"+dl,cmd) gs=re.search("s"+dl+"([^"+dl+"]*)"+dl+"([^"+dl+"]*)"+dl,cmd)
if not gs: else:
print("invalid sed search-and-replace pattern")
return (0,0)
s,r = gs.group(1),gs.group(2)
#print("search <{}> and replace <{}>".format(s,r))
sp=re.compile(s)
if op=='X' or op=='x':
dl=cmd[1]
gs=re.search("[xX]"+dl+"([^"+dl+"]*)"+dl,cmd) gs=re.search("[xX]"+dl+"([^"+dl+"]*)"+dl,cmd)
if not gs: if not gs:
print("invalid sed search pattern") print("invalid sed search pattern")
return (0,0) return 0,0
sp=re.compile(gs.group(1)) if op=='s':
ss,r = gs.group(1),gs.group(2)
#print("search <{}> and replace <{}>".format(s,r))
else:
ss=gs.group(1)
#print("search <{}>".format(s))
sp=re.compile(ss)
extra=g.group(4)[1:] + '\n' extra=a.group(4)[1:] + '\n'
try: try:
os.rename(filename,filename+'.'+bak_ext) os.rename(filename,filename+bak_ext)
except: except:
print("problem with filename; backup failed; no changes made") print("problem with filename; backup failed; no changes made")
return (0,0) return
i=h=0 i=h=0
try: try:
with open(filename+'.'+bak_ext) as d: with open(filename+bak_ext) as f:
with open(filename,'w') as f: with open(filename,'w') as g:
for lin in d: for lin in f:
i=i+1 i=i+1
m=(i>=start and i<=end) m=(i>=s and i<=e)
if op=='s' and m: if op=='s' and m:
lin=lin[:-1]
if sp.search(lin): h+=1 if sp.search(lin): h+=1
lin=sp.sub(r,lin) lin=sp.sub(r,lin)+'\n'
if op=='d' and m: if op=='d' and m:
h+=1 h+=1
continue # delete line continue # delete line
if op=='i' and m: if op=='i' and m:
#print("insert a line before {} <{}>".format(i,extra)) #print("insert a line before {} <{}>".format(i,extra))
f.write(extra) g.write(extra)
h+=1 h+=1
if op in "aids": if op in "aids":
f.write(lin) g.write(lin)
elif (m and (op=='x' and sp.search(lin)) or (op=='X' and not sp.search(lin))): elif m and (op=='x' if sp.search(lin) else op=='X'):
f.write(lin) g.write(lin)
h+=1 h+=1
if op=='a' and m: if op=='a' and m:
#print("append a line after {} <{}>".format(i,extra)) #print("append a line after {} <{}>".format(i,extra))
f.write(extra) g.write(extra)
h+=1 h+=1
#f.write("--file modifed by sed()--\n") #f.write("--file modifed by sed()--\n")
except OSError: except OSError:
@@ -125,31 +131,37 @@ def sed(filename, sed_cmd, bak_ext="bak"):
print("problem with the regex; try a different pattern") print("problem with the regex; try a different pattern")
return (i, h) return (i, h)
def _dir(d=''): def _dir(d='.'):
try: try:
for g in os.listdir(d): for f in os.listdir(d):
s=os.stat(d+'/'+g) s=os.stat(d+'/'+f)
print("{}rwx all {:9d} {}".format('d' if (s[0] & 0x4000) else '-',s[6],g)) print("{}rwx all {:9d} {}".format('d' if (s[0] & 0x4000) else '-',s[6],f))
except: except:
print("not a valid directory") print("not a valid directory")
s=os.statvfs('/') s=os.statvfs('/')
print("disk size:{:8d} KB disk free: {} KB\n".format(s[0]*s[2]//1024,s[0]*s[3]//1024)) print("disk size:{:8d} KB disk free: {} KB".format(s[0]*s[2]//1024,s[0]*s[3]//1024))
'''-----cut here if you only need the functions-----'''
def ext_cmd(a):
return
if 'tf_extend.py' in os.listdir():
import tf_extend
ext_cmd=tf_extend.cmd
'''-----cut here if you only need the above functions-----'''
def _help(): def _help():
print("simple shell v1.0") print("==Simple shell v1.1")
print(" cp/copy <src-file> <dest-file>") print(" cp/copy <src-file> <dest-file>")
print(" mv/move <src-file> <dest-file> rm/del <file>") print(" mv/move <src-file> <dest-file> \t\trm/del <file>")
print(" cd [<folder>] mkdir <folder> rmdir <folder>") print(" cd [<folder>] mkdir <folder>\t\trmdir <folder>")
print(" dir/ls [<folder>]") print(" dir/ls [<folder>]")
print(" cat/list [-n] [-l <n>,<m>] <file>") print(" cat/list [-n] [-l <n>,<m>] <file>")
print(" grep <pattern> <file>") print(" grep <pattern> <file>")
print(" sed <pattern> <file>") print(" sed <pattern> <file>")
print(" where <pattern> is '[<n>,<m>] s/search/replace/' or '<n>[,<m>]d' or '<n>i<text>' or '<n>a<text' ") print(" pattern is '<line-range><op><extra>' e.g'a/search/replace/', 'x!TODO:!', '43,49d', '8itext'")
print(" patterns with spaces require single-quotes sed ops are one of s/d/i/a/x/X")
print(" sed does not work across line boundaries sed s/x/X-patterns: non-/ delimiters are allowed")
print("file names must NOT have embedded spaces options must be early on the command line") print("file names must NOT have embedded spaces options must be early on the command line")
print("search patterns with spaces require single-quotes sed implements s/d/i/a/x/X") ext_cmd('help')
print("sed does not work across line boundaries sed s-patterns: non-/ delimiters are allowed")
def parseQuotedArgs(st): def parseQuotedArgs(st):
if st[0]=="'": if st[0]=="'":
@@ -162,19 +174,19 @@ def parseQuotedArgs(st):
return st.split()[0] return st.split()[0]
def main(): def main():
print("simple shell: cp/copy mv/move rm/del cat/list cd dir/ls mkdir rmdir grep sed help") print("Simple shell: cp/copy mv/move rm/del cat/list cd dir/ls mkdir rmdir grep sed help")
while 1: while 1:
numbers=False numbers=False
r=input(os.getcwd()+"$ ") r=input(os.getcwd()+"$ ")
rp=r.split() rp=r.split()
if not len(rp): continue if not len(rp): continue
op=rp[0] op=rp[0]
if op=='dir' or op=='ls': if op in ('dir','ls'):
_dir(rp[1] if len(rp)>1 else '') _dir(rp[1] if len(rp)>1 else '.')
elif op=='cat' or op=='list': elif op in ('cat','list'):
n=(" -n " in r) #print line-nums n=(" -n " in r) #print line-nums
s,e=(1,1000000) #start/end s,e=1,1000000 #start/end
g=re.search("\s+(-l\s*(\d+)([-,](\d+|\$)?)?)\s+",r[3:]) g=re.search("\s(-l\s*(\d+)([-,](\d+|\$)?)?)\s+",r[3:])
if g: if g:
s=e=int(g.group(2)) s=e=int(g.group(2))
if g.group(3): if g.group(3):
@@ -189,28 +201,32 @@ def main():
if len(rp)<3: if len(rp)<3:
print("sed pattern filename") print("sed pattern filename")
continue continue
lines, hits = sed(rp[-1],parseQuotedArgs(r[4:])) r=sed(rp[-1],parseQuotedArgs(r[4:]))
print("Lines processed: {} Lines modifed: {}".format(lines, hits)) if r:
print("Lines processed: {} Lines modifed: {}".format(*r))
elif op=='cd': elif op=='cd':
os.chdir(rp[1] if len(rp)>1 else '/') os.chdir(rp[1] if len(rp)>1 else '/')
elif op=='help': elif op=='help':
_help() _help()
ext_cmd(rp)
elif ext_cmd(rp):
pass
else: else:
try: try:
if op=='cp' or op=='copy': if op in ('cp','copy'):
cp(rp[1],rp[2]) cp(rp[1],rp[2])
elif op=='mkdir': elif op=='mkdir':
os.mkdir(rp[1]) os.mkdir(rp[1])
elif op=='rmdir': elif op=='rmdir':
os.rmdir(rp[1]) os.rmdir(rp[1])
elif op=='mv' or op=='move': elif op in('mv','move'):
os.rename(rp[1],rp[2]) os.rename(rp[1],rp[2])
elif op=='rm' or op=='del': elif op in('rm','del'):
os.remove(rp[1]) os.remove(rp[1])
else: else:
print("command not implemented") print("command not implemented")
except IndexError: except IndexError:
print("not enough argments; check syntax") print("not enough argments; check syntax with 'help'")
except OSError: except OSError:
print("file not found") print("file not found")
gc.collect() gc.collect()

62
tf_extend.py Normal file
View File

@@ -0,0 +1,62 @@
import os,network,socket,time,machine,gc
def cmd(args):
if args[0]=='ifconfig':
ifc=network.WLAN().ifconfig()
print("IP: {}\tmask: {}\tgateway: {}\tDNS: {}".format(*ifc))
return True
elif args[0]=='host':
if len(args)<2:
print("syntax: host <domain.name>")
return False
print("host <{}> is at {}".format(args[1],socket.getaddrinfo(args[1],80)[0][-1][0]))
return True
elif args[0]=='connect':
if len(args)<3:
print("syntax: connect <ssid> <password>")
return False
w=network.WLAN(network.STA_IF)
w.connect(args[1],args[2])
print("connecting...",end=' ')
time.sleep(3)
print(w.ifconfig() if w.isconnected() else "not yet connected; try 'ifconfig' in a few seconds")
return True
elif args[0]=='scan':
w=network.WLAN(network.STA_IF)
print("scanning...")
s=w.scan()
if len(s)==0:
print("no AP found")
return True
for i in s:
print("ch: {}\tRSSI: {}\t{}\tSSID: {}".format(i[2],i[3],"open" if i[4]==0 else "",i[0]))
return True
elif args[0]=='freq':
if len(args)==1 or args[1] in ("160","80"):
if len(args)>1:
machine.freq(int(args[1])*1000000)
print("master cpu frequency {}MHz".format(machine.freq()//1000000))
else:
print("syntax: freq [ 160 | 80 ]")
return True
elif args[0]=='exec':
if len(args)<2:
print("syntax: exec <python-filename>")
else:
try:
exec(open(args[1]).read(),globals(),globals())
except OSError:
print("file not found")
return True
elif args[0]=='free':
print("memory used: {}\tmemory free:{}".format(gc.mem_alloc(),gc.mem_free()))
return True
elif args[0]=='help':
print("==Extended commands")
print(" connect <essid> <password> \tscan")
print(" ifconfig \thost <domain.name>")
print(" freq [ 160 | 80 ] \texec <python-filename>")
print(" free")
return True
else:
return False