Cyoc down

[godleydemon]

WELL, I significantly overestimated how many entries cyoc/web archive had. It finished up sometime this morning since I was running the scans over night. Only ended up with 207k entries. I need to do some cleanup and future proofing then add author searching to the viewer but technically I'm done with my 1.0 version. It takes close to about 6 minutes to finish a full search. Which I'm not exactly happy about, so I'm going to be learning more about search and how I can speed it up on large databases. I'm probably going to have to swing away from sqlite, we'll see. But I've at least got most of what I need to start getting things done.

I'll try getting 1.0 out by the end of the week. Like I said, I want to do a bit of tweaking before I release anything. I'll also be releasing the source code of the scanner and the viewer so I want to get rid of the spaghetti nature, add in some comments here and there and generally just make it easier to read for anyone interested. Before releasing 1.0 I also want to add in the ability to change a bit of the look & feel of it. Adjusting things like fonts, font sizes and what's displayed in the top panel. So still some work to go pictured below is the current version, so it'll change a little. I was trying to avoid a menu system, but looks like I might have to include it, we'll see.



as a side note, I just noticed a bug where the total chapters are counting what's displayed in the top panel instead of the total rows in the db lol.

[godleydemon]

Originally Posted by lmorgado

Well, giving that it is taking so long for CYOC to fix the matter, the chances of it coming back are getting slimmer and slimmer. Godley, can you find my interactive stories and send them to me here. Think it's under my author name of LuisJM. If not, provide me the Wayback link with the CYOC site and I'll find them myself.

So I'm happy to say, I've found 401 entries under the author name "LuisJM". I used your name as a lab rat to make sure search was working for authors lol. I'm going to try and have 1.0 out tonight. I have some ideas to speed up search before I release. Give me a bit to work out a couple of bugs as well and I SHOULD have something up by tonight. The park has been rather busy with the state fair and stuff going on, so it's a bit busy

[godleydemon]

sorry I'm a little late. Work got busy and I couldn't spend as much time as I thought I would be able to. So here's the zip: https://mega.nz/file/aZkVwQ6B#v2A0nI...tOvOCkt-A8fsPw

So, a few things. I added a drop down box to select, none, synonyms and author. To switch between what mode of search your using. Synonyms works off of the synonyms.json file, something you can edit in and outside the program. Please be mindful, if the program crashes on startup after editing the json file, then most likely you have a syntax error somewhere. Not enough spaces, forgot a comma, forgot a quotation mark, ect. search works on keyword terms, if you select none in the drop down. Then it's going to search whatever is in that text box as a full phrase or word. So if you say type in "breasts shrinking" then it's going to search exactly "breasts shrinking" but if you type in "breasts, shrinking" then it's going to search breasts and shrinking. If you choose synonyms in the drop down, it works sorta the same way. If your search term matches the name of one of the arrays in the json file. Then it'll search for everything inside of that array. So searching for "breasts shrinking" is going to search about 1000+ different permutations of that. If you type in "breasts, shrinking" it's going to search with breasts synonyms and shrinking syonyms and display both. I originally had search setup to where both search terms had to be present in the contents of the row to display in the top panel. But I changed search today a crap ton to try and make it faster and I think I borked that part of it. So it's just going to show rows that have either breasts synonyms or shrinking synonyms or both.

The program might hang on first startup or when first selecting a database. I think I got that figured out and it'll no longer happen, but it might, so forwarning. It's just because the databases are fucking huge and take a bit to load. When they do load, they do not appear in one lump sum like I had it before. The top panel will still be blank until you search for something. This was to try and speed things up and free up memory.

Good news! I got synonym searches down from like 10 minutes down to about 30 seconds to a minute. The ETA loading bar isn't exactly accurate since it doesn't really have time to figure out what the hell to display. none and author search still take a little while. I haven't transferred them over to the new functions to do the quick searching. I ran out of time and got to tired before heading home (already stayed an hour over so far). I've gotten search pretty much done beyond that and a few bugs here and there. I also need to do a new scan on changing mirror to grab authors and to make sure I grabbed everything. I did something different with changing mirror and tried design a spider bot to web scrape everything, but I've been staring at this code for a while and I think I wrote this while stoned. The only other problem that's not easily solved and would have to be something I fix through a rescan. It did not save new line paragraphs from the original scan of data. So all of Cyoc's contents are just one paragraph. Unless the paragraphs were separated by a blank new line. This is annoying and I've thought of thousands of ways I might be able to fix it, but it all comes down to, the easiest would just be a rescan. Which I might still do, who knows. Because I'll have to do a rescan anyways to grab branch data in later builds. I had planned to add in a look and feel system to at least adjust font sizes. I also planned to add in functionality to select and copy things but completely spaced it and forgot about it till I started typing this.

To be honest, the .exe is completely untested on any other machine but this one. So I have no idea if it's going to actually work or not. I normally like testing stuff like this on several computers first before I do any kind of distribution or further testing. I always tend to screw exe's up for some reason -shrugs- the zip file includes both databases for cyoc and changing mirror, the synonym.json file and of course the exe and the internal files it needs to function. Hopefully everything works right, let me know if it doesn't and I'll beat it over the head with a brick. Thanks for waiting guys

[lmorgado]

Nice of you use my name to help get the filters going. I am hoping to add more stuff in your CYOC site. I just hope that I can upload more stories in the future.

[OhZone]

Avast didn't like the viewer.exe. Oh, well.

[godleydemon]

Originally Posted by OhZone

Avast didn't like the viewer.exe. Oh, well.

can you tell me what it marked it as? might give me a clue on what in the world it's even seeing as being bad. I think I'll throw my own exe into a tester real fast to see what happens.

--edit--

Okay, I figured out what happened here. Avast picks it up as "Win64:Malware-gen" which doesn't tell me alot, Avast has always been kind of vague. But thankfully Microsoft was a little more specific "Program:Win32/Wacapew.C!ml" which gave me a starting point. Turns out, these virus scanners just flag every exe built with pyinstaller. I wrote this all in python and have used pyinstaller before to create my exe's, but I haven't built one in a long while. For whatever reason, virus scanners just flag everything pyinstaller creates. Forcing you to put in for an exception. I'll see what other software is out there so I can build the exe with something that's not going to flag virus scanners.

In case anyone else is wondering or curious. Below are the ones that flagged it and what they see it as.

9/68 security vendors flagged this file as malicious.
Avast: Win64:Malware-gen
AVG: Win64:Malware-gen
DeepInstinct: MALICIOUS
Jiangmin: Trojan.Generic.hunjo
McAfee Scanner: Ti!003B547761AE
Microsoft: Program:Win32/Wacapew.C!ml
SecureAge: Malicious
SentinelOne (Static ML): Static AI - Suspicious PE
Trellix ENS: Artemis!E2680A7DB178

Now this is a reminder. Download things off the internet at your own risk, always have a virus scanner or something between you and what your downloading. My program isn't a virus or malware, but it's my word against the world so to speak and you make your own decisions based on what you see. If you choose to download my program, it'll work and do everything I set it up to do (hopefully, because I'm sure there are some bugs in there lol). It'll probably flag but you can flag it for an exception if you want to run it. I promise it won't do anything malicious, but if anyone still feels uncomfortable with that. They can wait for a few days and I'll try and have a version out that doesn't use pyinstaller and won't flag. I'll try and get a download up of just the databases as well, so you don't even have to use my program if you don't want to. You'll just have to make your own or use a database tool.

[godleydemon]

Quick update. I sent the exe into various antivirus sites for review. Mcafee already responded and should have updated there virus database. So if you keep your database up to date then it'll pass it's inspection now.

Next I started working on a few of the features I wanted to include. I've been severly annoyed that I grabbed the contents of the site as a string. Stripping all of it's formatting before stuffing it into the sqlite database. I added in a function, that regrabs the contents of whatever you click on in the viewer. Just to grab it's formatting data. It'll then update that entry in the db so it'll never have to grab it again.

Now for something more fun and slightly more interesting. At least visually. I moved the branches button up top and to the right. It made more sense up there anyhow since it's related searching. In it's place, I added a look & feel button to change how the preview panel is displayed. Making it significantly easier to read now and change it's style. Now, I no longer have to squint lol I included a picture in the attachments showing the preview panel working with that feature.

Don't know when I'll get this new version done, I want to add in a section to handle branching. So you can still following story threads.