lyrics: Avoid scraping non-lyrics from the Genius backend

[the-confessor]

Is there any practical way to debug the lyrics plugin?

I'm seen an unusual behavior with a particular release, I don't know if it is more wide-spread or not, trying to dig into it to find out.

I have the lyrics plugin enabled in my beets configuration, with all the default lyrics plugin settings.

I'm importing Queer (Remixes) by release ID. All of the other tags look correct, but the lyrics tag ends up capturing the track listing of another Garbage release these tracks happen to appear on.

I've tried manually searching the various lyrics search engines that the documentation indicates beets relies on. Two of them return the correct lyrics, the rest have no match, none of them have the below content.

Manually using the lyrics plugin also reproduces the issue.

What would sensible next steps be to debug this?

$ beet lyrics Garbage Queer Danny Saber mix -f -p
lyrics: fetched lyrics: Garbage - Queer - Queer (Danny Saber mix)
Album Art Tracklist Disc 1: Album + B-sides

01. "Supervixen"
02. "Queer"
03. "Only Happy When It Rains"
04. "As Heaven Is Wide"
05. "Not My Idea"
06. "A Stroke Of Luck"
07. "Vow"
08. "Stupid Girl"
09. "Dog New Tricks"
10. "My Lover's Box"
11. "Fix Me Now"
12. "Milk"
13. "Subhuman"
14. "Girl Don't Come"
15. "Sleep"
16 "Driving Lesson"
17 "Trip My Wire"
18 "#1 Crush" (Nellee Hooper Mix)
19 "Butterfly Collector"
20. "Alien Sex Fiend"
21. "Kick My Ass"

Disc 2: "Stupid Girl" + "Queer" remixes

01 "Stupid Girl" (Danny Saber Remix)
02 "Stupid Girl" (Rabbit In The Moon - Future Retro Mix)
03 "Stupid Girl" (Jason Bentley & Bruno Guez Shoegazer Mix)
04 "Stupid Girl" (Dreadzone Vocal Mix)
05 "Stupid Girl" (Dreadzone Dub Version)
06 "Stupid Girl" (Red Snapper Mix)
07 "Stupid Girl" (Todd Terry Tee's Radio Mix)
08 "Stupid Girl" (Todd Terry Tee's Freeze Club)
09 "Stupid Girl" (Todd Terry Tee's In-House Dub)
10 "Stupid Girl" (Todd Terry's Bonus Beats)
11 "Queer" (Danny Saber Alt Vox Mix)
12 "Queer" (Rabbit In The Moon Hefty Bag Mix)
13 "Queer" (Rabbit In The Moon F.T.O.I. AKA Dub Mix)
14 "Queer" (Adrian Sherwood The Very Queer Dub Bin Mix)
15 "Queer" (Martin Gore & Paul Freegard The Most Beautiful Woman in Town Mix)

Disc 3: "Milk" + extra remixes

01. "Milk" (Siren Mix)
02. "Milk" (The Wicked Mix) [feat. Tricky]
03. "Milk" (Goldie Completely Trashed Mix)
04. "Milk" (Rabbit In The Moon Tricky Mix) [feat. Tricky]
05. "Milk" (Rabbit In The Moon Got It Mix)
06. "Milk" (Rabbit In The Moon Butchered Vegas Mix)
07. "Milk" (Rabbit In The Moon Udder Edit)
08. "Milk" (Goldie VIP Rufige Trash Your Shit Mix)
09. "Milk" (Massive Attack Trance Mix)
10. "Milk" (Massive Attack Classic Mix)
11. "Milk" (Massive Attack Primal Mix)
12. "Milk" (Massive Attack Ultra Classic Mix)
13. "Milk" (Massive Attack D Mix)
14. "Dog New Tricks" (The Pal Mix)
15. "#1 Crush" (Original Mix)

Disc 4: Alternative versions

01. "Vow" (Joan of Ark Mix)
02. "Vow" (Stripped Down Mix)
03. "Vow" (Tuesday Night Club Mix)
04. "Fix Me Now" (Alternate Subbass Mix)
05. "Subhuman" (Supersize Mix)
06. "#1 Crush" (Early Demo Mix)
07. "A Stroke of Luck" (Early Demo Mix)
08. "My Lover's Box" (Early Demo Mix)
09. "Not My Idea" (Early Demo Mix)
10. "Only Happy When It Rains" (Early Demo Mix)
11. "Supervixen" (Early Demo Mix)
lyrics: fetched lyrics: Garbage - Queer (Remixes) - Queer (Danny Saber mix)
Album Art Tracklist Disc 1: Album + B-sides

01. "Supervixen"
02. "Queer"
03. "Only Happy When It Rains"
04. "As Heaven Is Wide"
05. "Not My Idea"
06. "A Stroke Of Luck"
07. "Vow"
08. "Stupid Girl"
09. "Dog New Tricks"
10. "My Lover's Box"
11. "Fix Me Now"
12. "Milk"
13. "Subhuman"
14. "Girl Don't Come"
15. "Sleep"
16 "Driving Lesson"
17 "Trip My Wire"
18 "#1 Crush" (Nellee Hooper Mix)
19 "Butterfly Collector"
20. "Alien Sex Fiend"
21. "Kick My Ass"

Disc 2: "Stupid Girl" + "Queer" remixes

01 "Stupid Girl" (Danny Saber Remix)
02 "Stupid Girl" (Rabbit In The Moon - Future Retro Mix)
03 "Stupid Girl" (Jason Bentley & Bruno Guez Shoegazer Mix)
04 "Stupid Girl" (Dreadzone Vocal Mix)
05 "Stupid Girl" (Dreadzone Dub Version)
06 "Stupid Girl" (Red Snapper Mix)
07 "Stupid Girl" (Todd Terry Tee's Radio Mix)
08 "Stupid Girl" (Todd Terry Tee's Freeze Club)
09 "Stupid Girl" (Todd Terry Tee's In-House Dub)
10 "Stupid Girl" (Todd Terry's Bonus Beats)
11 "Queer" (Danny Saber Alt Vox Mix)
12 "Queer" (Rabbit In The Moon Hefty Bag Mix)
13 "Queer" (Rabbit In The Moon F.T.O.I. AKA Dub Mix)
14 "Queer" (Adrian Sherwood The Very Queer Dub Bin Mix)
15 "Queer" (Martin Gore & Paul Freegard The Most Beautiful Woman in Town Mix)

Disc 3: "Milk" + extra remixes

01. "Milk" (Siren Mix)
02. "Milk" (The Wicked Mix) [feat. Tricky]
03. "Milk" (Goldie Completely Trashed Mix)
04. "Milk" (Rabbit In The Moon Tricky Mix) [feat. Tricky]
05. "Milk" (Rabbit In The Moon Got It Mix)
06. "Milk" (Rabbit In The Moon Butchered Vegas Mix)
07. "Milk" (Rabbit In The Moon Udder Edit)
08. "Milk" (Goldie VIP Rufige Trash Your Shit Mix)
09. "Milk" (Massive Attack Trance Mix)
10. "Milk" (Massive Attack Classic Mix)
11. "Milk" (Massive Attack Primal Mix)
12. "Milk" (Massive Attack Ultra Classic Mix)
13. "Milk" (Massive Attack D Mix)
14. "Dog New Tricks" (The Pal Mix)
15. "#1 Crush" (Original Mix)

Disc 4: Alternative versions

01. "Vow" (Joan of Ark Mix)
02. "Vow" (Stripped Down Mix)
03. "Vow" (Tuesday Night Club Mix)
04. "Fix Me Now" (Alternate Subbass Mix)
05. "Subhuman" (Supersize Mix)
06. "#1 Crush" (Early Demo Mix)
07. "A Stroke of Luck" (Early Demo Mix)
08. "My Lover's Box" (Early Demo Mix)
09. "Not My Idea" (Early Demo Mix)
10. "Only Happy When It Rains" (Early Demo Mix)
11. "Supervixen" (Early Demo Mix)

[the-confessor]

I came across another example. (Output truncated for sanity.)

$ beet lyrics -f -p Clint Eastwood Ed Case Refix edit
lyrics: fetched lyrics: Gorillaz - Clint Eastwood - Clint Eastwood (Ed Case Refix edit)
1. "Tomorrow Comes Today"
2. "Clint Eastwood" (featuring Del the Funky Homosapien)
3. "19-2000"
4. "Rock the House" (featuring Del the Funky Homosapien)
5. "Feel Good Inc." (single edit) (featuring De La Soul)
...

[wisp3rwind]

Please run beets in verbose mode, i.e. beet -vv lyrics Garbage Queer Danny Saber mix -f -p, that should make the lyrics plugin (and all other components of beets) log a lot more details about what they do --- That's also why the issue template here on Github says to do so. Also, please indicate how you configured the lyrics plugin (or, better, append the full configuration, as is asked for in the issue template).

[the-confessor]

Huh... I managed to bypass the template somehow.

The -vv argument gave me the info I need. Looks like Genius is the culprit. (I thought I tried a manual search there earlier, but I guess I missed it.)

Interesting that it is searching Genius even though the documentation suggests Genius is not in the default list.

So is the content on Genius wrong, or is beets reading from it incorrectly? The Genius UI is pretty awful so it's hard to tell what the raw data might look like. It appears that when I search for the individual track it re-routes me to a page with release details, not a track lyrics page.

$ beet -vv lyrics Garbage Queer Danny Saber mix -f -p
user configuration: /home/ed/.config/beets/config.yaml
data directory: /home/ed/.config/beets
plugin paths: 
Sending event: pluginload
lyrics: Disabling google source: no API key configured.
library database: /media/music/musiclibrary.db
library directory: /media/music/purchased
Sending event: library_opened
lyrics: failed to fetch: http://lyrics.wikia.com/Garbage:Queer_%28Danny_Saber_Mix%29 (410)
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/Garbage/Queer-%28Danny-Saber-Mix%29 (404)
lyrics: got lyrics from backend: Genius
lyrics: fetched lyrics: Garbage - Queer - Queer (Danny Saber mix)
Sending event: write
open failed: [Errno 2] No such file or directory: b'/media/music/Garbage - Queer/Garbage - Queer - 04 - Queer (Danny Saber mix).flac'
error reading /media/music/Garbage - Queer/Garbage - Queer - 04 - Queer (Danny Saber mix).flac: [Errno 2] No such file or directory: b'/media/music/Garbage - Queer/Garbage - Queer - 04 - Queer (Danny Saber mix).flac'
Sending event: database_change
Album Art Tracklist Disc 1: Album + B-sides

01. "Supervixen"
02. "Queer"
03. "Only Happy When It Rains"
04. "As Heaven Is Wide"
05. "Not My Idea"
06. "A Stroke Of Luck"
07. "Vow"
08. "Stupid Girl"
09. "Dog New Tricks"
10. "My Lover's Box"
11. "Fix Me Now"
12. "Milk"
13. "Subhuman"
14. "Girl Don't Come"
15. "Sleep"
16 "Driving Lesson"
17 "Trip My Wire"
18 "#1 Crush" (Nellee Hooper Mix)
19 "Butterfly Collector"
20. "Alien Sex Fiend"
21. "Kick My Ass"

Disc 2: "Stupid Girl" + "Queer" remixes

01 "Stupid Girl" (Danny Saber Remix)
02 "Stupid Girl" (Rabbit In The Moon - Future Retro Mix)
03 "Stupid Girl" (Jason Bentley & Bruno Guez Shoegazer Mix)
04 "Stupid Girl" (Dreadzone Vocal Mix)
05 "Stupid Girl" (Dreadzone Dub Version)
06 "Stupid Girl" (Red Snapper Mix)
07 "Stupid Girl" (Todd Terry Tee's Radio Mix)
08 "Stupid Girl" (Todd Terry Tee's Freeze Club)
09 "Stupid Girl" (Todd Terry Tee's In-House Dub)
10 "Stupid Girl" (Todd Terry's Bonus Beats)
11 "Queer" (Danny Saber Alt Vox Mix)
12 "Queer" (Rabbit In The Moon Hefty Bag Mix)
13 "Queer" (Rabbit In The Moon F.T.O.I. AKA Dub Mix)
14 "Queer" (Adrian Sherwood The Very Queer Dub Bin Mix)
15 "Queer" (Martin Gore & Paul Freegard The Most Beautiful Woman in Town Mix)

Disc 3: "Milk" + extra remixes

01. "Milk" (Siren Mix)
02. "Milk" (The Wicked Mix) [feat. Tricky]
03. "Milk" (Goldie Completely Trashed Mix)
04. "Milk" (Rabbit In The Moon Tricky Mix) [feat. Tricky]
05. "Milk" (Rabbit In The Moon Got It Mix)
06. "Milk" (Rabbit In The Moon Butchered Vegas Mix)
07. "Milk" (Rabbit In The Moon Udder Edit)
08. "Milk" (Goldie VIP Rufige Trash Your Shit Mix)
09. "Milk" (Massive Attack Trance Mix)
10. "Milk" (Massive Attack Classic Mix)
11. "Milk" (Massive Attack Primal Mix)
12. "Milk" (Massive Attack Ultra Classic Mix)
13. "Milk" (Massive Attack D Mix)
14. "Dog New Tricks" (The Pal Mix)
15. "#1 Crush" (Original Mix)

Disc 4: Alternative versions

01. "Vow" (Joan of Ark Mix)
02. "Vow" (Stripped Down Mix)
03. "Vow" (Tuesday Night Club Mix)
04. "Fix Me Now" (Alternate Subbass Mix)
05. "Subhuman" (Supersize Mix)
06. "#1 Crush" (Early Demo Mix)
07. "A Stroke of Luck" (Early Demo Mix)
08. "My Lover's Box" (Early Demo Mix)
09. "Not My Idea" (Early Demo Mix)
10. "Only Happy When It Rains" (Early Demo Mix)
11. "Supervixen" (Early Demo Mix)
lyrics: failed to fetch: http://lyrics.wikia.com/Garbage:Queer_%28Danny_Saber_Mix%29 (410)
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/Garbage/Queer-%28Danny-Saber-Mix%29 (404)
lyrics: got lyrics from backend: Genius
lyrics: fetched lyrics: Garbage - Queer (Remixes) - Queer (Danny Saber mix)
Sending event: write
Sending event: after_write
Sending event: database_change
Album Art Tracklist Disc 1: Album + B-sides

01. "Supervixen"
02. "Queer"
03. "Only Happy When It Rains"
04. "As Heaven Is Wide"
05. "Not My Idea"
06. "A Stroke Of Luck"
07. "Vow"
08. "Stupid Girl"
09. "Dog New Tricks"
10. "My Lover's Box"
11. "Fix Me Now"
12. "Milk"
13. "Subhuman"
14. "Girl Don't Come"
15. "Sleep"
16 "Driving Lesson"
17 "Trip My Wire"
18 "#1 Crush" (Nellee Hooper Mix)
19 "Butterfly Collector"
20. "Alien Sex Fiend"
21. "Kick My Ass"

Disc 2: "Stupid Girl" + "Queer" remixes

01 "Stupid Girl" (Danny Saber Remix)
02 "Stupid Girl" (Rabbit In The Moon - Future Retro Mix)
03 "Stupid Girl" (Jason Bentley & Bruno Guez Shoegazer Mix)
04 "Stupid Girl" (Dreadzone Vocal Mix)
05 "Stupid Girl" (Dreadzone Dub Version)
06 "Stupid Girl" (Red Snapper Mix)
07 "Stupid Girl" (Todd Terry Tee's Radio Mix)
08 "Stupid Girl" (Todd Terry Tee's Freeze Club)
09 "Stupid Girl" (Todd Terry Tee's In-House Dub)
10 "Stupid Girl" (Todd Terry's Bonus Beats)
11 "Queer" (Danny Saber Alt Vox Mix)
12 "Queer" (Rabbit In The Moon Hefty Bag Mix)
13 "Queer" (Rabbit In The Moon F.T.O.I. AKA Dub Mix)
14 "Queer" (Adrian Sherwood The Very Queer Dub Bin Mix)
15 "Queer" (Martin Gore & Paul Freegard The Most Beautiful Woman in Town Mix)

Disc 3: "Milk" + extra remixes

01. "Milk" (Siren Mix)
02. "Milk" (The Wicked Mix) [feat. Tricky]
03. "Milk" (Goldie Completely Trashed Mix)
04. "Milk" (Rabbit In The Moon Tricky Mix) [feat. Tricky]
05. "Milk" (Rabbit In The Moon Got It Mix)
06. "Milk" (Rabbit In The Moon Butchered Vegas Mix)
07. "Milk" (Rabbit In The Moon Udder Edit)
08. "Milk" (Goldie VIP Rufige Trash Your Shit Mix)
09. "Milk" (Massive Attack Trance Mix)
10. "Milk" (Massive Attack Classic Mix)
11. "Milk" (Massive Attack Primal Mix)
12. "Milk" (Massive Attack Ultra Classic Mix)
13. "Milk" (Massive Attack D Mix)
14. "Dog New Tricks" (The Pal Mix)
15. "#1 Crush" (Original Mix)

Disc 4: Alternative versions

01. "Vow" (Joan of Ark Mix)
02. "Vow" (Stripped Down Mix)
03. "Vow" (Tuesday Night Club Mix)
04. "Fix Me Now" (Alternate Subbass Mix)
05. "Subhuman" (Supersize Mix)
06. "#1 Crush" (Early Demo Mix)
07. "A Stroke of Luck" (Early Demo Mix)
08. "My Lover's Box" (Early Demo Mix)
09. "Not My Idea" (Early Demo Mix)
10. "Only Happy When It Rains" (Early Demo Mix)
11. "Supervixen" (Early Demo Mix)
Sending event: cli_exit

directory: /media/music/purchased
library: /media/music/musiclibrary.db
per_disc_numbering: yes
import:
    move: no
    copy: yes
    write: yes
    log: beetslog.txt
art_filename: cover
plugins: fetchart lyrics fromfilename
paths:
    default: $albumartist - $album/$albumartist - $album - $track - $title
    singleton: $artist/$artist - $title
    comp: $album%aunique{}/$album%aunique{} - $track - $artist - $title
ui:
    color: yes
discogs:
    user_token: (token removed)

[njelich]

I just found it crashing while searching tekstowo.pl (I'm on afc0728).

These are the relevant lines of error.

    return html.find("div", class_="song-text").get_text()
AttributeError: 'NoneType' object has no attribute 'get_text'

I also printed the html to the console, but I am unsure why the error is appearing. The div with the class song-text seems to exist and I've never worked on beets before. Could it have something to do with there being two divs with the class?

I uploaded the html, but had to do it with txt extension due to GitHub limitations.

temp.txt

[wisp3rwind]

I just found it crashing while searching tekstowo.pl

That sounds very much like a different problem, could you open a separate issue to track it? @xhocquet that's probably your turf.

[wisp3rwind]

The -vv argument gave me the info I need. Looks like Genius is the culprit. (I thought I tried a manual search there earlier, but I guess I missed it.)

Interesting that it is searching Genius even though the documentation suggests Genius is not in the default list.

So is the content on Genius wrong, or is beets reading from it incorrectly? The Genius UI is pretty awful so it's hard to tell what the raw data might look like. It appears that when I search for the individual track it re-routes me to a page with release details, not a track lyrics page.

Looks like the Genius plugin needs something like the bad_triggers mechanism that's implemented in the Google backend:

beets/beetsplug/lyrics.py

Lines 558 to 568 in 2fa2d1b

	bad_triggers = ['lyrics', 'copyright', 'property', 'links']
	if artist:
	bad_triggers += [artist]
	for item in bad_triggers:
	bad_triggers_occ += [item] * len(re.findall(r'\W%s\W' % item,
	text, re.I))
	if bad_triggers_occ:
	self._log.debug(u'Bad triggers detected: {0}', bad_triggers_occ)
	return len(bad_triggers_occ) < 2

[wisp3rwind]

Actually, the issue appears to be different: Genius used to use song pages with "Tracklist + album art" in the title, which is now deprecated, https://genius.com/Genius-how-to-annotate-artwork-on-genius-annotated,

Additionally, NEVER create a [Tracklist + Album Art] page.

but old pages like this are still around. We should filter such results.

I came across another example. (Output truncated for sanity.)

$ beet lyrics -f -p Clint Eastwood Ed Case Refix edit
lyrics: fetched lyrics: Gorillaz - Clint Eastwood - Clint Eastwood (Ed Case Refix edit)
1. "Tomorrow Comes Today"
2. "Clint Eastwood" (featuring Del the Funky Homosapien)
3. "19-2000"
4. "Rock the House" (featuring Del the Funky Homosapien)
5. "Feel Good Inc." (single edit) (featuring De La Soul)
...

This looks a little different, but the website for this specific song seems to be deleted now, so I can't really check.

[wisp3rwind]

curl \
    -H "Authorization: Bearer <genius_api_key from lyrics.py>"
    -H "User-Agent: beets/1.5.0"
    --get https://api.genius.com/search
    -d q=garbage

yields

{
    "error":"invalid_token",
    "error_description":"The access token provided is expired, revoked, malformed or invalid for other reasons."
}

Am I missing something, or has the key indeed expired?

[sampsyo]

Hmm, that's odd—it did work here when I constructed the HTTPie invocation myself:

$ http get 'https://api.genius.com/search?q=garbage' Authorization:'Bearer <redacted>' User-Agent:'beets/1.5.0'

In case it's relevant, the key is fairly long and split across two lines in the source code, which is sort of error prone for manual copying/pasting…