Automation by dannyjacobs · Pull Request #3 · HERA-Team/heratape

dannyjacobs · 2025-04-06T23:58:41Z

I think I have a working system. Could use some review to make sure its readable and sane. @bhazelton and maybe @plaplant
Next step is testing in screen mode. Once we get through a couple tapes we'll move to a deaemon.

enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # # On branch automation # Changes to be committed: # new file: ../src/heratape/tape_system.py # # Untracked files: # ../heratape.yml

adding help draft notes to readme

codecov · 2025-04-06T23:59:07Z

Codecov Report

❌ Patch coverage is 20.68966% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.42%. Comparing base (357aa2f) to head (9151422).

Files with missing lines	Patch %	Lines
src/heratape/files.py	23.52%	13 Missing ⚠️
src/heratape/tapes.py	16.66%	10 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##              main       #3      +/-   ##
===========================================
- Coverage   100.00%   93.42%   -6.58%     
===========================================
  Files            5        5              
  Lines          336      365      +29     
  Branches        74       78       +4     
===========================================
+ Hits           336      341       +5     
- Misses           0       24      +24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jaguirre · 2025-04-07T16:36:01Z

If you really want other people to be able to run this, adding to the README documentation or adding another bit of docs that explain in English what the workflow is - what scripts, in what order, need to be run for both normal operation and fixing problems - will be huge. Also, for my old brain, it's really helpful to know the order of doing things for reading all this OOP code which is never sequential / chronological. (Just wait, your brain will slow down too.)

Roughing in more about how it all works.

dannyjacobs · 2025-04-07T23:12:19Z

Judging by how many dozens of hours of work it took to get to this point, this brain is already well into senescence. Have added more detail. Please ask obvious questions so I don't have to write a bunch of detail you don't care about.

…ation

bhazelton

High-level comment: Putting console scripts in a folder at the top-level called "scripts" is not the recommended way to set up console scripts. The right way to do this is to write functions in .py files in the package folder (src/heratape) and then set up entry points in the pyproject.toml. See here: https://setuptools.pypa.io/en/latest/userguide/entry_point.html

That puts the code in a place that can actually be tested.
I'm happy to help with this if you want.

Second high-level comment: we need unit tests.

Third high-level comment: please install pre-commit and let it reformat your code to match the style. That will make it easier for me to read.

bhazelton · 2025-04-08T16:52:05Z

+Thge changer is 
+`/dev/sg25`


trails off...

thats telling you the changer device path. What else should it say?

bhazelton · 2025-04-08T16:54:33Z

+    # list the names of files with no write date
+    # set tape_id =None to list all, or to a tape serial number to list those
+    # return a list of files that weren't signed off with a write date.
+    #  this can happen if the write was interrupted before the db could be informed of success
+    #  writes can take hours, so this unhappy situation is likely


need a proper docstring, numpy style.

I think the comment above is actually wrong in addition to not being nicely formatted. It says this returns a list. It looks to me like it returns a tuple of lists

bhazelton · 2025-04-08T16:57:26Z

+    #  writes can take hours, so this unhappy situation is likely
+    with HTSessionWrapper(testing=testing) as ht_sess: 
+        if tape_id is None:
+            return ht_sess.query(func.distinct(Files.jd)).where(Files.write_date == None).all(), \


This can be written instead as:

Suggested change

return ht_sess.query(func.distinct(Files.jd)).where(Files.write_date == None).all(), \

return ht_sess.query(Files.jd).distinct().where(Files.write_date == None).all(), \

which I find slightly easier to read.

This also applies 2 lines down

bhazelton · 2025-04-08T16:57:53Z

+    #input: a tape serial number 
+    #output: total data volume in bytes


need a proper docstring, numpy style

I also think that I'd expect to find this in tapes.py rather than files.py.

it operates on the files table, not tapes.

bhazelton · 2025-04-08T20:43:26Z

+    #input drivenum = 0 or 1                                                                                                                                                                                                                            
+    #returns: VolumeTag of tape in drive                                                                                                                                                                                                                


need a proper docstring, numpy style

bhazelton · 2025-04-08T21:19:17Z

+    else:
+        logger.info(" TESTING MODE: skipping real tar to tape")
+        logger.debug(f' the tar command: time tar -cf /dev/nst{mydrive} {len(files)} not printed')
+    if True:


why bother with this if statement?

the debug log tells you the tar command that would have been run. But doesn't actually do it.

bhazelton · 2025-04-08T21:22:11Z

+# first list all the tapes in the DB with incomplete writes. Hopefully FEW
+tape_jds = {}
+with HTSessionWrapper(testing=TESTING) as ht_sess: 
+    unfinished_tapes = ht_sess.query(func.distinct(Files.tape_id)).where(Files.write_date == None).all()


Suggested change

unfinished_tapes = ht_sess.query(func.distinct(Files.tape_id)).where(Files.write_date == None).all()

unfinished_tapes = ht_sess.query(Files.tape_id).distinct().where(Files.write_date == None).all()

bhazelton · 2025-04-08T21:22:55Z

+        unfinished_jds = ht_sess.query(func.distinct(Files.jd)).\
+        where(Files.write_date == None).where(Files.tape_id == utape[0]).all() 


Suggested change

unfinished_jds = ht_sess.query(func.distinct(Files.jd)).\

where(Files.write_date == None).where(Files.tape_id == utape[0]).all()

unfinished_jds = ht_sess.query(Files.jd).distinct().where(

Files.write_date == None).where(Files.tape_id == utape[0]

).all()

bhazelton · 2025-04-08T21:24:44Z

+    for utape in unfinished_tapes:
+        unfinished_jds = ht_sess.query(func.distinct(Files.jd)).\
+        where(Files.write_date == None).where(Files.tape_id == utape[0]).all() 
+        unfinished_jds = [ujd[0] for ujd in unfinished_jds] #why do I get a list of length 1 tuples??


I'm not sure. It'd be easier to figure out if there was a test. I can try to write one, but if you have some test code lying around please add it!

there is no unsubmitted code. This is it!

bhazelton · 2025-04-08T21:27:01Z

+from heratape.tapes import list_tapes_in_db, add_new_tapes_to_db
+
+update_tapes(testing=TESTING)
+#def add_new_tapes_to_db(tapes,testing=TESTING):


It doesn't look like this file is doing much...

isn't this what you want scripts to be? just a function with no code in the actual script?

dannyjacobs · 2025-04-10T19:24:36Z

@bhazelton RE Testing
How do we test functions that use the tape hardware?

…guments error)

…ording to their position in the cassette at selection time

for more information, see https://pre-commit.ci

Danny Jacobs and others added 8 commits April 2, 2025 10:09

adding incompleteness and tape usage tabulation

2230a47

file functions

6b9aa56

tape functions

a80f800

adding main backup process

96300c7

adding tape scanner

735cc0d

adding script to recover from a failed write

55e278e

Update README.md

8300145

adding help draft notes to readme

dannyjacobs requested a review from bhazelton April 6, 2025 23:58

Update README.md

54f7cec

Roughing in more about how it all works.

Danny Jacobs added 3 commits April 7, 2025 17:20

adding env yaml

dc8bd73

Merge branch 'automation' of github.com:HERA-Team/heratape into autom…

1880846

…ation

add backup priorities for hera

4b226cb

bhazelton reviewed Apr 8, 2025

View reviewed changes

Danny Jacobs added 12 commits April 10, 2025 13:43

point at github for librarian

3825c5b

colon

d98a138

send tar a file of filenames instead of arguments. (fixes too many ar…

10d5f95

…guments error)

fix too many argument error in retry script

9037331

fix copypasta bs

1127c17

tapeid

bc6b1c4

filename

6420030

unfinished

5fba380

unfinished_

c87da7d

DOTAPE=False in testing

70a6992

filename again

4042edf

fixes from first hot tape test

30b5cb4

Danny Jacobs and others added 26 commits April 12, 2025 10:56

empyslot

9dfa865

DOTAPE True

dd32fb3

add check drive online

6bcc967

update_tape_usage

dfd957a

moving update_tape_usage

d62f0b2

add check for tape to finishh calibrating

592867d

fix indentation error

f4979a8

not defined

cca1338

list_tapes

90eea5e

Decimal??

a6dc65f

fix newtapeid not found

a7ccdd2

fix sleep not defined

151e74b

add jd to list_tapes

a3b8355

add a session

abbfa3a

use imported Session obj

e71eca5

TESTING

d3a2866

how do sessions work

2740691

Files

4adfe6a

==

8e0283a

None session

748fa8e

better docstring

09cda79

make sure subprocesses exits with an error when if tar errors

8c16390

logs and comments now correctly explain that drives process tapes acc…

ef5bd2c

…ording to their position in the cassette at selection time

replace popen with error trapping call, fix Empty bug

c6c0d0d

fixes empty drive issue

085f889

[pre-commit.ci] auto fixes from pre-commit.com hooks

9151422

for more information, see https://pre-commit.ci

	return ht_sess.query(func.distinct(Files.jd)).where(Files.write_date == None).all(), \
	return ht_sess.query(Files.jd).distinct().where(Files.write_date == None).all(), \

		#input: a tape serial number
		#output: total data volume in bytes

		#input drivenum = 0 or 1
		#returns: VolumeTag of tape in drive

	unfinished_tapes = ht_sess.query(func.distinct(Files.tape_id)).where(Files.write_date == None).all()
	unfinished_tapes = ht_sess.query(Files.tape_id).distinct().where(Files.write_date == None).all()

		unfinished_jds = ht_sess.query(func.distinct(Files.jd)).\
		where(Files.write_date == None).where(Files.tape_id == utape[0]).all()

		Thge changer is
		`/dev/sg25`

Uh oh!

Conversation

dannyjacobs commented Apr 6, 2025

Uh oh!

codecov Bot commented Apr 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jaguirre commented Apr 7, 2025

Uh oh!

dannyjacobs commented Apr 7, 2025

Uh oh!

bhazelton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dannyjacobs commented Apr 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Apr 6, 2025 •

edited

Loading