New options for keeping images #10

JulianSauer · 2018-05-23T13:56:35Z

I've added options to keep tags if they match a regular expression or if they were created after a specified date. E.g.:
./cleanreg.py -r http://192.168.56.2:5000 -n mysql -k 5 -i -re .*release.* -d 01.01.2018
will keep the 5 latest but also tags containing the word "release" as well as all those created since 01.01.2018.

Configs have to be rewritten to match the additional parameters with the option to use an underscore to ignore the parameter:
<repo> <k> <re> <d>

E.g.: mysql 5 _ _ keeps the 5 latest and does not consider regular expressions or dates.

kekru · 2018-05-31T19:35:05Z

Hi Julian,

thanks for your contribution. Here are a few thought of mine, but Halil has to decide ;)

The option to filter by a tag name is a good idea. But I would prefer to name the option --tagname instead of --regex, because it checks the tagnames.

But maybe we dont need an additional flag at all. Lets say we would allow --reponame to contain a ":", e.g --reponame postgres:9.4-alpine. The same way as you would write it in a dockerfile.

In cleanreg.py we check if there is a ":" in the reponame. If yes, we need to check the tagname.

Then we could add a --regex flag which stands for "enable interpreting of image and tag names as regex".

I think, if we add more features to the configfile, then we should better switch to a json file. The underscores are not very readable.

And maybe we need to change the testframework, when we add more and more tests. I think, it was not my best idea to use shunit :D It's a bit tricky

kekru · 2018-05-31T19:37:55Z

cleanreg.py

+    # check if date is valid
+    if args.date is not None:
+        try:
+            datetime.strptime(args.date, '%d.%m.%Y')


This should better be an ISO date format, maybe allow day (2018-05-31) and datetime (2018-05-31T18:03:24+00:00)

kekru · 2018-05-31T19:38:36Z

cleanreg.py

@@ -55,6 +56,8 @@ def parse_arguments():
    parser.add_argument('-k', '--keepimages', help="Amount of images (not tags!) which should be kept "
                                                   "for the given repo (if -n is set) or for each repo of the "
                                                   "registry (if -cf is set).", type=int)
+    parser.add_argument('-re', '--regex', help="Image tags matching the regular expression will be kept", default=None)


see my other comment. I think this flag should only enable regex and not define the tagname. The tagname can be part of of the reponame, e.g postgres:3.6 where postgres is the image name and 3.6 the tag

kekru · 2018-05-31T19:39:31Z

cleanreg.py

@@ -55,6 +56,8 @@ def parse_arguments():
    parser.add_argument('-k', '--keepimages', help="Amount of images (not tags!) which should be kept "
                                                   "for the given repo (if -n is set) or for each repo of the "
                                                   "registry (if -cf is set).", type=int)
+    parser.add_argument('-re', '--regex', help="Image tags matching the regular expression will be kept", default=None)
+    parser.add_argument('-d', '--date', help="Keep images which were created since this date. Format: dd.mm.yyyy", default=None)


better --after, then we could also add a --before

kekru · 2018-05-31T19:41:55Z

cleanreg.py

+        if date is not None and date != "_" and date != "":
+            parsed_date = datetime.strptime(date, '%d.%m.%Y')
+            for tag in deletion_tags.keys():
+                tag_date = datetime.strptime(deletion_tags[tag]['date'].split('T')[0], '%Y-%m-%d')


Better interpret the full datetime string (with the T). You can search for ISO Datetime

kekru · 2018-05-31T19:42:37Z

cleanreg.py

@@ -597,13 +632,13 @@ def get_deletiontags(verbose, tags_dates_digests, repo, repo_count):

    repo_del_tags = {}
    repo_del_digests = {}
-    for repo, count in repos_counts.iteritems():
+    for repo, (count, regex, date) in repos_counts.iteritems():


regex should better be tagname

hcguersoy · 2018-06-08T10:43:33Z

Sorry coming again so late (just again not get pinged by Github...).
I'll check this.

hcguersoy · 2018-06-21T08:58:48Z

PTAL @JulianSauer

JulianSauer · 2018-07-19T16:53:34Z

Hey,

I've added most of the suggestions from @kekru:

A tagname is now an additional part of the -n option which can be added; if -re is used the tag will be parsed as a regular expression
Date format changed to ISO 8601
The time of day can additionally be specified: YYYY-MM-DDThh:mm:ss
I've changed the parameter name from --date to --since because it will keep all images that were created since then (and not just after)
Configs are now yaml files

kekru · 2018-07-27T18:48:11Z

cleanreg.py

@@ -89,8 +110,9 @@ def parse_arguments():
        parser.error("[-n] and [-cf] cant be used together")

    # hackish dependent arguments
-    if (bool(args.reponame) or args.clean_full_catalog) ^ (args.keepimages is not None):
-        parser.error("[-n] or [-cf] have to be used together with [-k].")
+    if (bool(args.reponame) or args.clean_full_catalog) ^ (args.keepimages is not 0 or args.regex is True or args.since is not None):


a small comment for this check and the check beneath would be nice, to better understand

kekru · 2018-07-27T19:02:21Z

cleanreg.py

+
+    deletion_tags = all_tags
+    if regex and tagname != "":
+        deletion_tags = {k: deletion_tags[k] for k in deletion_tags if re.match(tagname, k)}


Crazy python syntax, what do I have to search for, to find an explanation? :)

kekru · 2018-07-27T19:09:41Z

cleanreg.py

+                try:
+                    parsed_date = datetime.strptime(args.since, '%Y%m%dT%H%M%S')
+                except ValueError:
+                    parsed_date = datetime.strptime(args.since, '%Y-%m-%dT%H:%M:%S')


the date parsing could be extracted to a function (returning the parsed date or None if not parsable), because you need it two times

kekru · 2018-07-27T19:22:28Z

cleanreg.py

+        if ammount_tags <= keep_count:
+            deletion_tags.clear()
+        else:
+            deletion_tags = collections.OrderedDict(islice(deletion_tags.iteritems(), ammount_tags - keep_count))


Ah comment would be great.
if amount_tags - keep_count is 7, then this will keep the first 7 in the list and remove the others. The first 7 are the oldest tags.
Right?

deletion_tags contains the tags that will be deleted. So it will remove the 7 oldest and keep the others.
(I'll add a comment)

JulianSauer · 2018-08-02T15:50:36Z

I've changed a few more things considering the suggestions of @kekru (Simplified a for loop, moved date parsing to a function and added comments).

hcguersoy · 2018-08-06T11:45:43Z

Looks fine, thank you. I'll add some additional stuff into the README before I create a new release.

JulianSauer added 15 commits April 23, 2018 15:24

Add options to filter for dates and regular expressions

1d84542

Fix date parsing

beb43d6

Update readme

dba7e4c

Add test for regex

605e8bd

Fix error when using -n without -k

709e3c7

Improve regex test and add test for date

f5a1649

Add config options for regex and date

df687e9

Fix parsing of parameters

613fc74

Add checks for empty repos

93fc41b

Add exception for empty regex

4eeaa84

Update example config

0c697d9

Add tests for configs with regex and date

5353297

Update tests cases

5b08825

Update readme

2744fb8

Code cleanup

534050a

kekru reviewed May 31, 2018

View reviewed changes

JulianSauer added 11 commits July 12, 2018 10:12

Fix date test

906c431

Use image:tag syntax

643edab

Change format

64a7e7c

Fix indent

a45a056

Fix regex test

8e61477

Fix cf test

864eed2

Update config check tests

f469830

Switch to yaml format for repo files

509aae2

Update tests for yaml configs

fbe0c7d

Update descriptions

b66fd54

Fix package in docker image

41d61a9

JulianSauer added 3 commits July 19, 2018 15:54

Rename parameter

616d18a

Update config tests

8a37dc4

Change date format

576e8d8

kekru reviewed Jul 27, 2018

View reviewed changes

JulianSauer added 4 commits August 2, 2018 16:09

Add comments

90a18b6

Typo

04b85cc

Move date parsing to separate function

3a6a201

Simplified loop syntax

8d31700

hcguersoy merged commit 5bb6518 into hcguersoy:master Aug 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New options for keeping images #10

New options for keeping images #10

JulianSauer commented May 23, 2018

kekru commented May 31, 2018

kekru May 31, 2018

kekru May 31, 2018

kekru May 31, 2018

kekru May 31, 2018

kekru May 31, 2018

hcguersoy commented Jun 8, 2018

hcguersoy commented Jun 21, 2018

JulianSauer commented Jul 19, 2018

kekru Jul 27, 2018

kekru Jul 27, 2018

kekru Jul 27, 2018

kekru Jul 27, 2018

JulianSauer Aug 2, 2018

JulianSauer commented Aug 2, 2018

hcguersoy commented Aug 6, 2018

New options for keeping images #10

New options for keeping images #10

Conversation

JulianSauer commented May 23, 2018

kekru commented May 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hcguersoy commented Jun 8, 2018

hcguersoy commented Jun 21, 2018

JulianSauer commented Jul 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JulianSauer commented Aug 2, 2018

hcguersoy commented Aug 6, 2018