patb/tf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

-add copyright

Browse Source 
-shorten variable names (for size)
-simplify sed() command scan
-sed backup extension includes the . dot now
-add extended commands in tf_extend.py
This commit is contained in:
Pat Beirne
2021-04-06 07:26:24 -04:00
parent 34f88c5948
commit 83857b5571
3 changed files with 209 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
README.md
View File

@@ -2,6 +2,8 @@
A module for manipulating files in the *MicroPython* environment.
[TOC]
## Oveview
I discovered *MicroPython* when working on the ESP8266 processor. Everything seemed very nice, except it was awkward moving files around. All the methods I could find required a back-and-forth with the programmer's desktop.
@@ -71,6 +73,8 @@ tf.cp('log.txt','log.bak')`
Simply copies a source file to a destination file. Filenames may include folders or . or .. prefixes. The destination is overwritten if it exists. This function reads-&-writes one line at a time, so it can handle megabyte files. Typical speeds are 100kB/sec on an ESP8266.
**NOTE** this function *only works on text files*. Line lengths of up to 4096 work fine on the ESP8266.
#### cat()
```
@@ -124,7 +128,7 @@ tf.grep('config.ini', '\[\w*\]', numbers = True)
#### sed()
```
sed(filename, pattern, bak_ext="bak")
sed(filename, pattern, bak_ext=".bak")
in: filename the file to edit
pattern a sed pattern, involving one of "aidsxX"
bak_ext the extension to use when creating the file backup (without the dot)
@@ -172,7 +176,9 @@ s/thier/their/
s@ratio\s*=\s*num/denom@ratio = num/denom if denom else 0@
```
**Note**: you will need some free space on your disk, the same size as the source file, as a backup file is *always* made. To edit an 800k file, you should have 800k of free space.
**Note**: The function version of sed() can have embedded space characters in the pattern; the command line version (below) requires single-quotes around patterns that have space characters.
**Note**: You will need some free space on your disk, the same size as the source file, as a backup file is *always* made. To edit an 800k file, you should have 800k of free space.
**Note**: The functions for
@@ -235,17 +241,18 @@ Commands with invalid syntax return a line of information, and are ignored. Non
In its present form, the module has these limitations:
* filenames are limited to 255 chars
* search patterns involving \ escapes may or may not work properly
* the esp8266 implementation does not allow \1,\2 type pattern substitution
* files must be text
* or have a `\n` at least every 4096 characters
* `sed()` requires lines <=2048 characters, and this `sed()` won't match binary chars
* search patterns involving \ escapes other than `\'` probably won't work
* in the simple shell
* filenames must not have spaces
* patterns with spaces ***must*** be quoted
* the target of `cp`and `mv` *cannot* be a simple a directory-name as in Linux; write the whole filename *w.r.t,* the current directory
* the complexity of pattern matching is limited.
* try to format the grep patterns so they avoid deep stack recursion. For example, '([^#]|\\#)\*' has a very generous search term as the first half, and can cause deep-stack recursion. The equivalent '(\\#|[^#]\*)' is more likely to succeed.
* with sed-search-and-replace, the parsed line *includes* the terminal \n, so if you replace the text all the way to the end of the line, you will delete the terminal \n and this line will merge with the subsequent line
* if the replacement is '', the line will appear to vanish, e.g. `s/^#.*//` will delete comment lines
* pattern matching to \n and \r does not work
* with sed, lines are parsed and saved one-line-at-a-time, so pattern matching to \n and \r does not work
* this simple shell is different than [mpfshell](https://github.com/wendlers/mpfshell) in that this shell runs entirely on the target device. There is no allowance in this shell for transferring files in/out of the target.
## Examples
@@ -276,14 +283,16 @@ Search a log file for an incident
[command line]
grep [Ee]rror log.txt
grep '2021-02-12 16:\d\d' log.txt
[search and keep a record ]
# search and keep a record
cp log.txt log.details
sed 'x/2021-02-12 16:\d\d` log.details
```
## Installation
Move the 'tf.py' file over to the target. You can use `webrepl` [command line program](https://github.com/micropython/webrepl) or the **WEBREPL** [web page](https://micropython.org/webrepl/) .
~If you need help with getting connected to your MicroPython board, there are excellent howto guides here and here~ TODO!
Move the 'tf.py' file over to the target. You can use `webrepl` [command line program](https://github.com/micropython/webrepl) or the **WEBREPL** [web page](http://micropython.org/webrepl/) . If you want the command line extensions, then send over the `tf_extend.py` file as well
Once the module is present in the file system of the target, you can use the **REPL** command line interface to invoke it
@@ -315,12 +324,30 @@ This is the *simple command line*. You can type `dir` to get an idea of what's a
/test_dir$
```
If you don't need the *simple command line*, you can still use the methods listed above. Feel free to cut the `tf.py` module in half by deleting everything below the line
If you don't need the *simple command line*, you can still use the methods listed at the top of this readme. Feel free to cut the `tf.py` module in half by deleting everything below the line
```
def help():
def ext_cmd(a):
```
## Extensions
I found the simple command line so useful, I added some extra non-file-related functions. These are included in the optional filr `tf_extend.py'. the available command list is extended to include
```
scan # scan and show the local AP's
connect essid password # create a persistent wifi connection
ifconfig # show current ip address
host <domain.name> # do an DNS lookup
freq [160 | 80] # get/set the ESP8266 frequency
exec <python-filename> # execute a small python file
free # display the heap size: used + free
```
The `tf.py` module checks to see if the `tf_extend.py` files exists, and forwards unknown commands to it. The `help` system also extends when the extension file exists.
Installing the extensions module uses about 2k of flash/disk space and 2kB of heap ram.
## Performance
Typical performance on an ESP8266 @80MHz, 90kB log file, 1200 lines; serial connected terminal @115200baud

180
tf.py
View File

@@ -1,122 +1,128 @@
# tf Text File manipulations
# for micropython and other tiny environments
#NOTE: the ESP8266 port cannot to \1,\2 type replacements in the s/search/replace/ operator
# (c) Pat Beirne patb@pbeirne.com
import re,os,sys,gc
def _file_scan(src,dest,start=1,end=0xFFFFFFFF,numbers=False,grep_func=None):
#src is a filename, dst is an open handle
# ====legend====
# e=end
# f=input file
# g=output file
# i=iteration variable
# m=line range
# r=replace text
# s=start/search/string
# sp=search regex
def transfer(src,dest,first=1,last=0xFFFFFFFF,numbers=False,grep_func=None):
#src is a filename, dst is a handle
i=0
try:
with open(src) as f:
for line in f:
for lin in f:
i=i+1
if i<start or i>end:
if i<first or i>last:
continue
if grep_func and not grep_func(line):
if grep_func and not grep_func(lin):
continue
if numbers:
dest.write(str(i)+' ')
dest.write(line)
dest.write(lin)
except:
print("could not open file {}".format(src))
def cp(src_f, dst_f):
def cp(src,dest):
try:
with open(dst_f,'w') as g:
_file_scan(src_f,g)
with open(dest,'w') as g:
transfer(src,g)
except:
print("could not write to file {}".format(dst_f))
print("could not write to file {}".format(dest))
def grep(filename, pattern, numbers=False):
m=re.compile(pattern)
if not m:
print("grep() called with invalid pattern")
return None
_file_scan(filename,sys.stdout,numbers=numbers,grep_func=(lambda x:m.search(x)))
print()
return
transfer(filename,sys.stdout,numbers=numbers,grep_func=(lambda x:m.search(x)))
def cat(filename, first=1, last=1000000, numbers=False, title=True):
if title:
print("===={}=====".format(filename))
_file_scan(filename,sys.stdout,first,last,numbers=numbers)
print()
transfer(filename,sys.stdout,first,last,numbers=numbers)
def sed(filename, sed_cmd, bak_ext="bak"):
#print("sed() called with sed_cmd=<{}>".format(sed_cmd))
def sed(filename, sed_cmd, bak_ext=".bak"):
# parse the sed_cmd
# group 1,3 are the n-start, n-end group 4 is command
g=re.search("^(\d*)([,-](\d+|\$))?\s*([sdaixX].*)",sed_cmd)
if not g:
print("sed() failed; 2nd argument must be a number followed by one of sdaixX; no changes applied")
return 0,0
cmd=g.group(4)
#print("sed() cmd parsed into <{}>,<{}> and <{}>".format(g.group(1),g.group(3),g.group(4)))
# group 1,3 are the n-start, n-end group 4 is command: aidsxX
a=re.search("^(\d*)([,-](\d+|\$))?\s*([sdaixX].*)",sed_cmd)
if not a:
print("sed() failed; 2nd argument must be a number-range followed by one of sdaixX; no changes applied")
return
cmd=a.group(4)
start,end=(1,1000000)
if g.group(1):
start=end=int(g.group(1))
if g.group(3):
end=1000000 if g.group(3)=='$' else int(g.group(3))
s,e=(1,1000000)
if a.group(1):
s=e=int(a.group(1))
if a.group(3):
e=1000000 if a.group(3)=='$' else int(a.group(3))
op=cmd[0]
if op not in "sdiaxX":
print("sed requires an operation, one of 's,d,i,a,x or X'")
return 0,0
return
#print("sed command parser of <{}> returned {} {} {} {}".format(cmd,sr,de,ins,add))
if op in "sxX" and len(cmd)<2:
if op in "sxX":
if len(cmd)<2:
print("invalid sed argument")
return (0,0)
return
dl=cmd[1]
if op=='s':
dl=cmd[1]
gs=re.search("s"+dl+"([^"+dl+"]*)"+dl+"([^"+dl+"]*)"+dl,cmd)
if not gs:
print("invalid sed search-and-replace pattern")
return (0,0)
s,r = gs.group(1),gs.group(2)
#print("search <{}> and replace <{}>".format(s,r))
sp=re.compile(s)
if op=='X' or op=='x':
dl=cmd[1]
else:
gs=re.search("[xX]"+dl+"([^"+dl+"]*)"+dl,cmd)
if not gs:
print("invalid sed search pattern")
return (0,0)
sp=re.compile(gs.group(1))
return 0,0
if op=='s':
ss,r = gs.group(1),gs.group(2)
#print("search <{}> and replace <{}>".format(s,r))
else:
ss=gs.group(1)
#print("search <{}>".format(s))
sp=re.compile(ss)
extra=g.group(4)[1:] + '\n'
extra=a.group(4)[1:] + '\n'
try:
os.rename(filename,filename+'.'+bak_ext)
os.rename(filename,filename+bak_ext)
except:
print("problem with filename; backup failed; no changes made")
return (0,0)
return
i=h=0
try:
with open(filename+'.'+bak_ext) as d:
with open(filename,'w') as f:
for lin in d:
with open(filename+bak_ext) as f:
with open(filename,'w') as g:
for lin in f:
i=i+1
m=(i>=start and i<=end)
m=(i>=s and i<=e)
if op=='s' and m:
lin=lin[:-1]
if sp.search(lin): h+=1
lin=sp.sub(r,lin)
lin=sp.sub(r,lin)+'\n'
if op=='d' and m:
h+=1
continue # delete line
if op=='i' and m:
#print("insert a line before {} <{}>".format(i,extra))
f.write(extra)
g.write(extra)
h+=1
if op in "aids":
f.write(lin)
elif (m and (op=='x' and sp.search(lin)) or (op=='X' and not sp.search(lin))):
f.write(lin)
g.write(lin)
elif m and (op=='x' if sp.search(lin) else op=='X'):
g.write(lin)
h+=1
if op=='a' and m:
#print("append a line after {} <{}>".format(i,extra))
f.write(extra)
g.write(extra)
h+=1
#f.write("--file modifed by sed()--\n")
except OSError:
@@ -125,31 +131,37 @@ def sed(filename, sed_cmd, bak_ext="bak"):
print("problem with the regex; try a different pattern")
return (i, h)
def _dir(d=''):
def _dir(d='.'):
try:
for g in os.listdir(d):
s=os.stat(d+'/'+g)
print("{}rwx all {:9d} {}".format('d' if (s[0] & 0x4000) else '-',s[6],g))
for f in os.listdir(d):
s=os.stat(d+'/'+f)
print("{}rwx all {:9d} {}".format('d' if (s[0] & 0x4000) else '-',s[6],f))
except:
print("not a valid directory")
s=os.statvfs('/')
print("disk size:{:8d} KB disk free: {} KB\n".format(s[0]*s[2]//1024,s[0]*s[3]//1024))
print("disk size:{:8d} KB disk free: {} KB".format(s[0]*s[2]//1024,s[0]*s[3]//1024))
'''-----cut here if you only need the functions-----'''
def ext_cmd(a):
return
if 'tf_extend.py' in os.listdir():
import tf_extend
ext_cmd=tf_extend.cmd
'''-----cut here if you only need the above functions-----'''
def _help():
print("simple shell v1.0")
print("==Simple shell v1.1")
print(" cp/copy <src-file> <dest-file>")
print(" mv/move <src-file> <dest-file> rm/del <file>")
print(" cd [<folder>] mkdir <folder> rmdir <folder>")
print(" mv/move <src-file> <dest-file> \t\trm/del <file>")
print(" cd [<folder>] mkdir <folder>\t\trmdir <folder>")
print(" dir/ls [<folder>]")
print(" cat/list [-n] [-l <n>,<m>] <file>")
print(" grep <pattern> <file>")
print(" sed <pattern> <file>")
print(" where <pattern> is '[<n>,<m>] s/search/replace/' or '<n>[,<m>]d' or '<n>i<text>' or '<n>a<text' ")
print(" pattern is '<line-range><op><extra>' e.g'a/search/replace/', 'x!TODO:!', '43,49d', '8itext'")
print(" patterns with spaces require single-quotes sed ops are one of s/d/i/a/x/X")
print(" sed does not work across line boundaries sed s/x/X-patterns: non-/ delimiters are allowed")
print("file names must NOT have embedded spaces options must be early on the command line")
print("search patterns with spaces require single-quotes sed implements s/d/i/a/x/X")
print("sed does not work across line boundaries sed s-patterns: non-/ delimiters are allowed")
ext_cmd('help')
def parseQuotedArgs(st):
if st[0]=="'":
@@ -162,19 +174,19 @@ def parseQuotedArgs(st):
return st.split()[0]
def main():
print("simple shell: cp/copy mv/move rm/del cat/list cd dir/ls mkdir rmdir grep sed help")
print("Simple shell: cp/copy mv/move rm/del cat/list cd dir/ls mkdir rmdir grep sed help")
while 1:
numbers=False
r=input(os.getcwd()+"$ ")
rp=r.split()
if not len(rp): continue
op=rp[0]
if op=='dir' or op=='ls':
_dir(rp[1] if len(rp)>1 else '')
elif op=='cat' or op=='list':
if op in ('dir','ls'):
_dir(rp[1] if len(rp)>1 else '.')
elif op in ('cat','list'):
n=(" -n " in r) #print line-nums
s,e=(1,1000000) #start/end
g=re.search("\s+(-l\s*(\d+)([-,](\d+|\$)?)?)\s+",r[3:])
s,e=1,1000000 #start/end
g=re.search("\s(-l\s*(\d+)([-,](\d+|\$)?)?)\s+",r[3:])
if g:
s=e=int(g.group(2))
if g.group(3):
@@ -189,28 +201,32 @@ def main():
if len(rp)<3:
print("sed pattern filename")
continue
lines, hits = sed(rp[-1],parseQuotedArgs(r[4:]))
print("Lines processed: {} Lines modifed: {}".format(lines, hits))
r=sed(rp[-1],parseQuotedArgs(r[4:]))
if r:
print("Lines processed: {} Lines modifed: {}".format(*r))
elif op=='cd':
os.chdir(rp[1] if len(rp)>1 else '/')
elif op=='help':
_help()
ext_cmd(rp)
elif ext_cmd(rp):
pass
else:
try:
if op=='cp' or op=='copy':
if op in ('cp','copy'):
cp(rp[1],rp[2])
elif op=='mkdir':
os.mkdir(rp[1])
elif op=='rmdir':
os.rmdir(rp[1])
elif op=='mv' or op=='move':
elif op in('mv','move'):
os.rename(rp[1],rp[2])
elif op=='rm' or op=='del':
elif op in('rm','del'):
os.remove(rp[1])
else:
print("command not implemented")
except IndexError:
print("not enough argments; check syntax")
print("not enough argments; check syntax with 'help'")
except OSError:
print("file not found")
gc.collect()

62
tf_extend.py Normal file
View File

@@ -0,0 +1,62 @@
import os,network,socket,time,machine,gc
def cmd(args):
if args[0]=='ifconfig':
ifc=network.WLAN().ifconfig()
print("IP: {}\tmask: {}\tgateway: {}\tDNS: {}".format(*ifc))
return True
elif args[0]=='host':
if len(args)<2:
print("syntax: host <domain.name>")
return False
print("host <{}> is at {}".format(args[1],socket.getaddrinfo(args[1],80)[0][-1][0]))
return True
elif args[0]=='connect':
if len(args)<3:
print("syntax: connect <ssid> <password>")
return False
w=network.WLAN(network.STA_IF)
w.connect(args[1],args[2])
print("connecting...",end=' ')
time.sleep(3)
print(w.ifconfig() if w.isconnected() else "not yet connected; try 'ifconfig' in a few seconds")
return True
elif args[0]=='scan':
w=network.WLAN(network.STA_IF)
print("scanning...")
s=w.scan()
if len(s)==0:
print("no AP found")
return True
for i in s:
print("ch: {}\tRSSI: {}\t{}\tSSID: {}".format(i[2],i[3],"open" if i[4]==0 else "",i[0]))
return True
elif args[0]=='freq':
if len(args)==1 or args[1] in ("160","80"):
if len(args)>1:
machine.freq(int(args[1])*1000000)
print("master cpu frequency {}MHz".format(machine.freq()//1000000))
else:
print("syntax: freq [ 160 | 80 ]")
return True
elif args[0]=='exec':
if len(args)<2:
print("syntax: exec <python-filename>")
else:
try:
exec(open(args[1]).read(),globals(),globals())
except OSError:
print("file not found")
return True
elif args[0]=='free':
print("memory used: {}\tmemory free:{}".format(gc.mem_alloc(),gc.mem_free()))
return True
elif args[0]=='help':
print("==Extended commands")
print(" connect <essid> <password> \tscan")
print(" ifconfig \thost <domain.name>")
print(" freq [ 160 | 80 ] \texec <python-filename>")
print(" free")
return True
else:
return False